{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Based Emotion Recognition with PyTorch\n",
    "\n",
    "![alt txt](img/emotion_classifier.png)\n",
    "\n",
    "In this notebook we are going to learn how to train deep neural networks, such as recurrent neural networks (RNNs), for addressing a natural language task known as **emotion recognition**. We will cover everything you need to know to get started with NLP using deep learning frameworks such as TensorFlow. We will cover the common best practices, functionalities, and steps you need to understand the basics of PyTorch APIs to build powerful predictive models via the computation graph. In the process of building our models, we will compare PyTorch and TensorFlow to let the learner appreciate the strenghts of each tool.\n",
    "\n",
    "by [Elvis Saravia](https://twitter.com/omarsar0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "1. Deep Learning Frameworks\n",
    "     - 1.1 Eager execution\n",
    "     - 1.2 Computation graph\n",
    "2. Tensors\n",
    "    - 2.1 Basic math with tensors\n",
    "    - 2.2 Transforming tensors\n",
    "3. Data\n",
    "    - 3.1 Preprocessing data\n",
    "        - Tokenization and Sampling\n",
    "        - Constructing Vocabulary and Index-Word Mapping\n",
    "    - 3.2 Converting data into tensors\n",
    "    - 3.3 Padding data\n",
    "    - 3.4 Binarization\n",
    "    - 3.5 Split data\n",
    "    - 3.6 Data Loader\n",
    "4. Model\n",
    "    - 4.1 Pretesting Model\n",
    "    - 4.2 Testing models with eager execution\n",
    "5. Training\n",
    "6. Evaluation on Testing Dataset\n",
    "    - 6.1 Confusion matrix\n",
    "- Final Words\n",
    "- References\n",
    "- *Storing models and setting checkpoints (Exercise)*\n",
    "- *Restoring models (Exercise)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Deep Learning Frameworks\n",
    "There are many deep learning frameworks such as Chainer, DyNet, MXNet, PyTorch, TensorFlow, and Keras. Each framework has their own strenghts which a researcher or a developer may want to consider before choosing the right framework. In my opinion, PyTorch is great for researchers and offers eager execution by default, but its high-level APIs require some understanding of deep learning concepts such as **affine layers** and **automatic differentiation**. On the other hand, TensorFlow was originally built as a low-level API that provides a robust list of functionalities to build deep learning models from the ground up. More recently, TensorFlow also offers **eager execution** and is equipped with a high-level API known as Keras.\n",
    "\n",
    "![alt txt](img/dl_frameworks.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Eager Execution\n",
    "Eager execution allows us to operate on the computation graph dynamically, also known as **imperative programming**. TensorFlow requires that you manually set this mode, while PyTorch comes with this mode by default. Below we import the necessary libraries to use PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.functional as F\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Computation Graph\n",
    "A simplified definition of a neural network is a string of functions that are **differentiable** and that we can combine together to get more complicated functions. An intuitive way to express this process is through computation graphs. \n",
    "\n",
    "![alt txt](http://colah.github.io/posts/2015-08-Backprop/img/tree-eval-derivs.png)\n",
    "\n",
    "Image credit: [Chris Olah](http://colah.github.io/posts/2015-08-Backprop/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Tensors\n",
    "Tensors are the fundamental data structure used to store data that will be fed as input to a computation graph for processing and applying tranformations. Let's create two tensors and multiply them, and then output the result. The figure below shows a 4-D Tensor.\n",
    "\n",
    "![alt txt](img/tensor.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 3.],\n",
      "        [3., 7.]])\n",
      "torch.Size([2, 2])\n"
     ]
    }
   ],
   "source": [
    "c = torch.tensor([[1.0, 2.0], [3.0, 4.0]])\n",
    "d = torch.tensor([[1.0, 1.0], [0.0, 1.0]])\n",
    "e = torch.matmul(c, d)\n",
    "print(e)\n",
    "print(c.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Math with Tensors\n",
    "PyTorch and other deep learning libraries like TensorFlow allow you to do **automatic differentation**. Let's try to compute the derivative of a function -- in this case that function is stored in the variable `z`. In PyTorch, the option `requires_grad=True` tracks all operations applied to the input tensor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(27., grad_fn=<MeanBackward1>)\n",
      "tensor([[4.5000, 4.5000],\n",
      "        [4.5000, 4.5000]])\n"
     ]
    }
   ],
   "source": [
    "### Automatic differentiation with PyTorch\n",
    "x = torch.ones(2, 2, requires_grad=True)\n",
    "\n",
    "# an operation of tensor\n",
    "y = x + 2 # y inherits grad_fn\n",
    "\n",
    "# apply operations on y\n",
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(out)\n",
    "\n",
    "out.backward()\n",
    "\n",
    "print(x.grad) # d(out)/dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can verfiy the output with the equations in the figure below:\n",
    "\n",
    "![alt txt](img/autograd.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Transforming Tensors\n",
    "We can also apply some transformation to a tensor such as adding a dimension or transposing it. Let's try both adding a dimension and transposing a matrix below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X shape:  torch.Size([2, 3])\n",
      "torch.Size([2, 1, 3])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[1, 4],\n",
       "        [2, 5],\n",
       "        [3, 6]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
    "print(\"X shape: \", x.size())\n",
    "\n",
    "# add dimension\n",
    "print(x.unsqueeze(1).size()) \n",
    "\n",
    "# transpose \n",
    "torch.transpose(x, 0,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Emotion Dataset\n",
    "In this notebook we are working on an emotion classification task. We are using the public emotion dataset provided [here](https://github.com/huseinzol05/NLP-Dataset/tree/master/emotion-english). The dataset contains tweets labeled into 6 categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import numpy as np\n",
    "import time\n",
    "import helpers.pickle_helpers as ph\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f8519d377b8>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEbCAYAAAAmmNiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAHEBJREFUeJzt3X2UHXWd5/H3x2SCoEBAelhMIomaQRGfMEJ2cXZY0BAEDaOoMGqyTiRnNIyO646AoxMPyh6fVnaYQcZAIsF1eBBnJAPBmEHR8SFAgwwxPJgmiiQLJhIEjwxCmM/+Ub+Gm04/VPredHW6P69z7umqX/3q1reS7v7cqvpVtWwTERFRx7OaLiAiIvYcCY2IiKgtoREREbUlNCIioraERkRE1JbQiIiI2hIaERFRW0IjIiJqS2hERERtCY2IiKhtYtMFdNpBBx3k6dOnN11GRMQe5dZbb/2V7a6h+o250Jg+fTrd3d1NlxERsUeRdF+dfjk9FRERtSU0IiKitoRGRETUltCIiIjaEhoREVFbQiMiImobMjQkLZe0RdJP+ln2YUmWdFCZl6QLJPVIukPSkS19F0jaUF4LWtpfI2ldWecCSSrtB0paU/qvkXRAZ3Y5IiKGq86RxqXA3L6NkqYBc4BftDSfCMwsr0XARaXvgcAS4GjgKGBJSwhcBJzRsl7vts4GbrA9E7ihzEdERIOGvLnP9vckTe9n0fnAR4BrWtrmAZfZNrBW0mRJhwDHAmtsbwOQtAaYK+lGYD/ba0v7ZcApwPXlvY4t77sCuBE4a5f2bhdMP/u63fXW/fr5p08a0e1FRHTCsK5pSJoHbLb9b30WTQHub5nfVNoGa9/UTzvAwbYfKNMPAgcPp9aIiOicXX6MiKR9gI9SnZoaEbYtyYPUtIjqdBgveMELRqqsiIhxZzhHGi8CZgD/JunnwFTgNkn/CdgMTGvpO7W0DdY+tZ92gF+WU1uUr1sGKsj2UtuzbM/q6hryeVsRETFMuxwattfZ/n3b021PpzqldKTtB4GVwPwyimo28Eg5xbQamCPpgHIBfA6wuix7VNLsMmpqPs9cI1kJ9I6yWsCO104iIqIBdYbcXg78CDhM0iZJCwfpvgrYCPQAFwPvBygXwD8J3FJe5/ZeFC99Linr3Et1ERzg08AbJG0AXl/mIyKiQXVGT50+xPLpLdMGFg/QbzmwvJ/2buCIftofAo4fqr6IiBg5uSM8IiJqS2hERERtCY2IiKgtoREREbUlNCIioraERkRE1JbQiIiI2hIaERFRW0IjIiJqS2hERERtCY2IiKgtoREREbUlNCIioraERkRE1JbQiIiI2hIaERFRW0IjIiJqS2hERERtCY2IiKgtoREREbUNGRqSlkvaIuknLW2fk3S3pDsk/ZOkyS3LzpHUI+keSSe0tM8tbT2Szm5pnyHpptJ+paRJpX2vMt9Tlk/v1E5HRMTw1DnSuBSY26dtDXCE7VcAPwXOAZB0OHAa8LKyzhclTZA0AbgQOBE4HDi99AX4DHC+7RcDDwMLS/tC4OHSfn7pFxERDZo4VAfb3+v7Kd/2t1pm1wKnlul5wBW2fwf8TFIPcFRZ1mN7I4CkK4B5ku4CjgP+pPRZAXwCuKi81ydK+9XA30mSbe/C/kWvT+w/wtt7ZGS3FxEjohPXNP4UuL5MTwHub1m2qbQN1P484Ne2t/dp3+G9yvJHSv+IiGhIW6Eh6a+A7cBXO1POsOtYJKlbUvfWrVubLCUiYkwbdmhI+u/AycA7W04ZbQamtXSbWtoGan8ImCxpYp/2Hd6rLN+/9N+J7aW2Z9me1dXVNdxdioiIIQwrNCTNBT4CvNn2Yy2LVgKnlZFPM4CZwM3ALcDMMlJqEtXF8pUlbL7DM9dEFgDXtLzXgjJ9KvDtXM+IiGjWkBfCJV0OHAscJGkTsIRqtNRewBpJAGtt/5nt9ZKuAu6kOm212PZT5X3OBFYDE4DltteXTZwFXCHpU8CPgWWlfRnwlXIxfRtV0ERERIPqjJ46vZ/mZf209fY/Dzivn/ZVwKp+2jfyzAir1vbHgbcNVV9ERIyc3BEeERG1JTQiIqK2hEZERNSW0IiIiNoSGhERUVtCIyIiaktoREREbQmNiIioLaERERG1JTQiIqK2hEZERNSW0IiIiNoSGhERUVtCIyIiaktoREREbQmNiIioLaERERG1JTQiIqK2hEZERNSW0IiIiNqGDA1JyyVtkfSTlrYDJa2RtKF8PaC0S9IFknok3SHpyJZ1FpT+GyQtaGl/jaR1ZZ0LJGmwbURERHPqHGlcCszt03Y2cIPtmcANZR7gRGBmeS0CLoIqAIAlwNHAUcCSlhC4CDijZb25Q2wjIiIaMmRo2P4esK1P8zxgRZleAZzS0n6ZK2uByZIOAU4A1tjeZvthYA0wtyzbz/Za2wYu6/Ne/W0jIiIaMtxrGgfbfqBMPwgcXKanAPe39NtU2gZr39RP+2DbiIiIhrR9IbwcIbgDtQx7G5IWSeqW1L1169bdWUpExLg23ND4ZTm1RPm6pbRvBqa19Jta2gZrn9pP+2Db2IntpbZn2Z7V1dU1zF2KiIihDDc0VgK9I6AWANe0tM8vo6hmA4+UU0yrgTmSDigXwOcAq8uyRyXNLqOm5vd5r/62ERERDZk4VAdJlwPHAgdJ2kQ1CurTwFWSFgL3AW8v3VcBbwR6gMeA9wDY3ibpk8Atpd+5tnsvrr+faoTW3sD15cUg24iIiIYMGRq2Tx9g0fH99DWweID3WQ4s76e9Gziin/aH+ttGREQ0J3eER0REbQmNiIioLaERERG1JTQiIqK2hEZERNSW0IiIiNoSGhERUVtCIyIiaktoREREbQmNiIioLaERERG1JTQiIqK2hEZERNSW0IiIiNoSGhERUVtCIyIiaktoREREbQmNiIioLaERERG1JTQiIqK2hEZERNTWVmhI+pCk9ZJ+IulySc+WNEPSTZJ6JF0paVLpu1eZ7ynLp7e8zzml/R5JJ7S0zy1tPZLObqfWiIho37BDQ9IU4APALNtHABOA04DPAOfbfjHwMLCwrLIQeLi0n1/6Ienwst7LgLnAFyVNkDQBuBA4ETgcOL30jYiIhrR7emoisLekicA+wAPAccDVZfkK4JQyPa/MU5YfL0ml/Qrbv7P9M6AHOKq8emxvtP0EcEXpGxERDRl2aNjeDHwe+AVVWDwC3Ar82vb20m0TMKVMTwHuL+tuL/2f19reZ52B2iMioiHtnJ46gOqT/wzg+cBzqE4vjThJiyR1S+reunVrEyVERIwL7Zyeej3wM9tbbT8J/CNwDDC5nK4CmApsLtObgWkAZfn+wEOt7X3WGah9J7aX2p5le1ZXV1cbuxQREYNpJzR+AcyWtE+5NnE8cCfwHeDU0mcBcE2ZXlnmKcu/bdul/bQyumoGMBO4GbgFmFlGY02iuli+so16IyKiTROH7tI/2zdJuhq4DdgO/BhYClwHXCHpU6VtWVllGfAVST3ANqoQwPZ6SVdRBc52YLHtpwAknQmsphqZtdz2+uHWGxER7Rt2aADYXgIs6dO8kWrkU9++jwNvG+B9zgPO66d9FbCqnRojIqJzckd4RETUltCIiIjaEhoREVFbQiMiImpLaERERG0JjYiIqC2hERERtSU0IiKitoRGRETUltCIiIjaEhoREVFbQiMiImpLaERERG0JjYiIqC2hERERtSU0IiKitoRGRETUltCIiIjaEhoREVFbQiMiImprKzQkTZZ0taS7Jd0l6T9LOlDSGkkbytcDSl9JukBSj6Q7JB3Z8j4LSv8Nkha0tL9G0rqyzgWS1E69ERHRnnaPNP4G+KbtlwCvBO4CzgZusD0TuKHMA5wIzCyvRcBFAJIOBJYARwNHAUt6g6b0OaNlvblt1hsREW0YdmhI2h/4r8AyANtP2P41MA9YUbqtAE4p0/OAy1xZC0yWdAhwArDG9jbbDwNrgLll2X6219o2cFnLe0VERAPaOdKYAWwFvizpx5IukfQc4GDbD5Q+DwIHl+kpwP0t628qbYO1b+qnPSIiGtJOaEwEjgQusv1q4Lc8cyoKgHKE4Da2UYukRZK6JXVv3bp1d28uImLcaic0NgGbbN9U5q+mCpFfllNLlK9byvLNwLSW9aeWtsHap/bTvhPbS23Psj2rq6urjV2KiIjBDDs0bD8I3C/psNJ0PHAnsBLoHQG1ALimTK8E5pdRVLOBR8pprNXAHEkHlAvgc4DVZdmjkmaXUVPzW94rIiIaMLHN9f8c+KqkScBG4D1UQXSVpIXAfcDbS99VwBuBHuCx0hfb2yR9Eril9DvX9rYy/X7gUmBv4PryioiIhrQVGrZvB2b1s+j4fvoaWDzA+ywHlvfT3g0c0U6NMT68fMXLR2xb6xasG7FtRYw2uSM8IiJqS2hERERtCY2IiKgtoREREbUlNCIioraERkRE1JbQiIiI2hIaERFRW0IjIiJqS2hERERtCY2IiKgtoREREbUlNCIiorZ2H40eEbvZXS956Yhu76V33zWi24s9S440IiKitoRGRETUltCIiIjaEhoREVFbQiMiImpLaERERG1th4akCZJ+LOnaMj9D0k2SeiRdKWlSad+rzPeU5dNb3uOc0n6PpBNa2ueWth5JZ7dba0REtKcTRxofBFoHdn8GON/2i4GHgYWlfSHwcGk/v/RD0uHAacDLgLnAF0sQTQAuBE4EDgdOL30jIqIhbYWGpKnAScAlZV7AccDVpcsK4JQyPa/MU5YfX/rPA66w/TvbPwN6gKPKq8f2RttPAFeUvhER0ZB2jzT+D/AR4D/K/POAX9veXuY3AVPK9BTgfoCy/JHS/+n2PusM1B4REQ0ZdmhIOhnYYvvWDtYz3FoWSeqW1L1169amy4mIGLPaOdI4BnizpJ9TnTo6DvgbYLKk3mdaTQU2l+nNwDSAsnx/4KHW9j7rDNS+E9tLbc+yPaurq6uNXYqIiMEMOzRsn2N7qu3pVBeyv237ncB3gFNLtwXANWV6ZZmnLP+2bZf208roqhnATOBm4BZgZhmNNalsY+Vw642IiPbtjqfcngVcIelTwI+BZaV9GfAVST3ANqoQwPZ6SVcBdwLbgcW2nwKQdCawGpgALLe9fjfUGxERNXUkNGzfCNxYpjdSjXzq2+dx4G0DrH8ecF4/7auAVZ2oMSIi2pc7wiMioraERkRE1JbQiIiI2vLnXiOiURf+2bdHdHuL//64Ed3eWJMjjYiIqC2hERERtSU0IiKitoRGRETUltCIiIjaEhoREVFbQiMiImpLaERERG0JjYiIqC2hERERtSU0IiKitoRGRETUltCIiIjaEhoREVFbQiMiImpLaERERG0JjYiIqG3YoSFpmqTvSLpT0npJHyztB0paI2lD+XpAaZekCyT1SLpD0pEt77Wg9N8gaUFL+2skrSvrXCBJ7exsRES0p50jje3Ah20fDswGFks6HDgbuMH2TOCGMg9wIjCzvBYBF0EVMsAS4GjgKGBJb9CUPme0rDe3jXojIqJNww4N2w/Yvq1M/wa4C5gCzANWlG4rgFPK9DzgMlfWApMlHQKcAKyxvc32w8AaYG5Ztp/ttbYNXNbyXhER0YCOXNOQNB14NXATcLDtB8qiB4GDy/QU4P6W1TaVtsHaN/XTHhERDWk7NCQ9F/g68Be2H21dVo4Q3O42atSwSFK3pO6tW7fu7s1FRIxbbYWGpN+jCoyv2v7H0vzLcmqJ8nVLad8MTGtZfWppG6x9aj/tO7G91PYs27O6urra2aWIiBhEO6OnBCwD7rL9hZZFK4HeEVALgGta2ueXUVSzgUfKaazVwBxJB5QL4HOA1WXZo5Jml23Nb3mviIhowMQ21j0GeDewTtLtpe2jwKeBqyQtBO4D3l6WrQLeCPQAjwHvAbC9TdIngVtKv3NtbyvT7wcuBfYGri+viIhoyLBDw/b3gYHumzi+n/4GFg/wXsuB5f20dwNHDLfGiIjorNwRHhERtSU0IiKitnauaURExBD+9ztOHtHtffjKa3fr++dIIyIiaktoREREbQmNiIioLaERERG1JTQiIqK2hEZERNSW0IiIiNoSGhERUVtCIyIiaktoREREbQmNiIioLaERERG1JTQiIqK2hEZERNSW0IiIiNoSGhERUVtCIyIiahv1oSFprqR7JPVIOrvpeiIixrNRHRqSJgAXAicChwOnSzq82aoiIsavUR0awFFAj+2Ntp8ArgDmNVxTRMS4NdpDYwpwf8v8ptIWERENkO2maxiQpFOBubbfW+bfDRxt+8w+/RYBi8rsYcA9I1jmQcCvRnB7I20s799Y3jfI/u3pRnr/DrXdNVSniSNRSRs2A9Na5qeWth3YXgosHamiWknqtj2riW2PhLG8f2N53yD7t6cbrfs32k9P3QLMlDRD0iTgNGBlwzVFRIxbo/pIw/Z2SWcCq4EJwHLb6xsuKyJi3BrVoQFgexWwquk6BtHIabERNJb3byzvG2T/9nSjcv9G9YXwiIgYXUb7NY2IiBhFEhoREVFbQmMXSXqTpPy77YFUmTZ0z4gYSH757bp3ABskfVbSS5ouZneSdICkVzRdR6e4uoA3mgdVtEXSBEl3N13H7ibpUEmvL9N7S9q36ZrGk4TGLrL9LuDVwL3ApZJ+JGnRWPnGlXSjpP0kHQjcBlws6QtN19VBt0l6bdNF7A62nwLukfSCpmvZXSSdAVwNfKk0TQW+0VxFnSPpYEnLJF1f5g+XtLDpuvpKaAyD7UepvnGvAA4B/pjql9GfN1pYZ+xf9u8twGW2jwZe33BNnXQ08CNJ90q6Q9I6SXc0XVQHHQCsl3SDpJW9r6aL6qDFwDHAowC2NwC/32hFnXMp1T1pzy/zPwX+orFqBjDq79MYbSS9GXgP8GLgMuAo21sk7QPcCfxtk/V1wERJhwBvB/6q6WJ2gxOaLmA3+3jTBexmv7P9hCQAJE0Exsp9AwfZvkrSOfD0zc1PNV1UXwmNXfdW4Hzb32tttP3YaDyUHIZzqT7tfN/2LZJeCGxouKaOsX2fpNcBM21/WVIX8Nym6+oU299tuobd7LuSPgrsLekNwPuBf264pk75raTnUUJQ0mzgkWZL2llu7hsGSQcDvefFb7a9pcl6oj5JS4BZwGG2/0DS84Gv2T6m4dI6ovyi+VvgpcAkqsfv/Nb2fo0W1iFl5OJCYA4gqg84l3gM/CKTdCTV/90RwE+ALuBU26Pq9GlCYxdJehvweeBGqm/aPwT+0vbVTdbVKZI+C3wK+Hfgm8ArgA/Z/r+NFtYhkm6nGshwm+1Xl7Y7bI+JUWKSuqke7Pk1qnCcD/yB7XMaLaxDJL0FuM7275quZXcop9sOo/rdco/tJxsuaSe5EL7rPga81vYC2/Op/rrgWDqPPKdcCD8Z+DnVtZu/bLSiznqifCrtPQXwnIbr6TjbPcAE20/Z/jIwt+maOuhNwE8lfUXSyeWX7JhQPpDuXR7KegpwZTn6GFUSGrvuWX1ORz3E2Pp37P0hPInqtM2oO6fapqskfQmYXIZv/gtwccM1ddJj5c8I3F7uJfoQY+j703bvIJSvAacD90q6pNmqOubjtn9TrrkdDywDLmq4pp2MmZQeQd+UtBq4vMyfBlzfYD2ddm25QezfgfeVC8WPN1xTx9j+fLmA+ijVaYC/tr2m4bI66d1UIXEm8CGqP2L21kYr6jDbT5Z7GQzsTfWp/L3NVtURvSOlTgIutn2dpE81WVB/ck1jGMp51d4Lp/9qe0zcXNSr3Nj3iO2nyumbfW0/2HRdUY+kvYEX2B7JP3s8IiSdSPVUhmOpriteBXzL9vYGy+oISddS/WXSNwBHUn1wu9n2KxstrI+ERk2Svm/7dZJ+Q/UJRy2L/wPYBnzO9hcbKbBDyv0m/4Pql84iSTOpRhpd23BpHdHy/9fqEaAb+LDtjSNfVedIehPVQI1JtmdIehVwru03N1xaR0i6HLgSuH6sXQwvP3tzgXW2N5T7pV5u+1sNl7aDhEaHlPHVP7R9WNO1tEPSlcCtwHzbR5Rv5B/aflXDpXWEpE8Cm4B/oAr+04AXUT0y5X22j22uuvZJuhU4DrixZXTYOtsvb7ayzhlrQ94l7Wf70XKEvxPb20a6psGMmQtkTbP9ENUh857uRbY/CzwJ1U2L7HhUtad7s+0v2f6N7UdtLwVOsH0l1SM49nRP9jN4Ycx8MiwjjG4G3kb11IKbJJ3abFVt+4fy9VaqI95bW17dTRU1kFwI7yDbDzRdQwc8Uc6J9w5JfREwlk4DPCbp7VTPDgM4lWcu9I+FX67rJf0JMKGcWvwA8MOGa+qk3iHvWwDKQI1/4Zn/zz2O7ZNVPRflj2z/oul6hpIjjehrCdVNfdMkfRW4AfhIsyV11DupRhhtAX5Zpt9VgvLMJgtrh6SvlMl7gZdRBf3lVKPERt1D79owJoe8l3uHrmu6jjpyTSN2Uq7PzKY6LbXW9q8aLimGIOlOqqcRXw/8t77LR9t58eGS9DmqpxT0Dnl/B3CH7bOaq6ozJK0A/s72LU3XMpiERuxE0hTgUFpOX/Z9QOOeqpzOOAOYzo7796dN1dQJkj4AvA94IdWwzacXUX2QfWEjhe0Gkt7KjkPe/6nJejql3B/1YuA+4Lc88383qh5xk9CIHUj6DNWnt/VUQ4mh+sYdK0M2fwj8K9VFxqcfO237640V1UGSLrL9vqbriF0n6dD+2m3fN9K1DCahETuQdA/wirE2Br6XpNvHyvDh8WSA+2vgmU/jY+UpvkcCr6Pa1x/Yvq3hknayx19Aio7bCPxe00XsRtdKemPTRcSusb2v7f36ee07hgLjr4EVwPOAg4AvS/pYs1XtLEcasQNJXwdeSTVq6umjDdsfaKyoDiqfWJ9DtW9PMsY+qcaeqxzlv9L242V+b+D20XbDcO7TiL5WlteYZHvfcuftTODZTdcT0eL/UX1P9t43tBc7DmoYFXKkEeOKpPcCHwSmArdTDS3+oe3jGy0sxj1J36B6PMoaqmsab6C6+30TjJ6j/YRGANXziRjkjujRNuxvuMp+vpbq/pNXSXoJ8L9sv6Xh0mKck7RgsOW2V4xULYPJ6anodXL5urh87b3D+F2Mjcdr9Hrc9uOSkLSX7bsljapzxjH+SJpA9Vcz39l0LUNJaATwzFhwSW/ofTpqcZak24Czm6ms4zZJmgx8A1gj6WGqm6kiGlP+ds2hkibZfqLpegaT0Ii+JOkY2z8oM/+FMTQ02/Yfl8lPSPoOsD/Vs7YimrYR+IGklVR3hANg+wvNlbSzhEb0tRBYLml/quGoDwN79CM2BmL7u03XENHi3vJ6FrBvw7UMKBfCo18lNOjnbzNExDiW0IidSDqJ6vHaT9/HYPvc5iqKGPvK6dKdfiHbPq6BcgaU01OxA0l/D+xD9XjtS6j+SNHNjRYVMT78z5bpZwNvBbY3VMuAcqQRO5B0h+1XtHx9LnC97T9suraI8UbSzbaParqOVjnSiL56H2HwmKTnA9uAQxqsJ2JcKI+36fUsYBbV6L5RJaERff1zuY/hc8BtVOdYL262pIhx4VaqnzdRPUzz51SjGUeVMTP+PjrmbuCp8keJLgTWUt0IFxG711nAq2zPoHoiw2+Bx5otaWcJjejr47Z/I+l1wHFUF8MvarimiPHgY7YfHe0/ewmN6Kv3T6CeBFxs+zpgUoP1RIwXe8TPXkIj+tos6UtUfyd8laS9yPdJxEjYI372MuQ2diBpH2AusM72BkmHAC+3/a2GS4sY0/aUn72ERkRE1DbqDn0iImL0SmhERERtCY2IiKgtoREREbUlNCIiorb/D0Uz5gCzSaONAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load data\n",
    "data = ph.load_from_pickle(directory=\"data/merged_training.pkl\")\n",
    "data.emotions.value_counts().plot.bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>emotions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>27383</th>\n",
       "      <td>i feel awful about it too because it s my job ...</td>\n",
       "      <td>sadness</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110083</th>\n",
       "      <td>im alone i feel awful</td>\n",
       "      <td>sadness</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140764</th>\n",
       "      <td>ive probably mentioned this before but i reall...</td>\n",
       "      <td>joy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100071</th>\n",
       "      <td>i was feeling a little low few days back</td>\n",
       "      <td>sadness</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>i beleive that i am much more sensitive to oth...</td>\n",
       "      <td>love</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18231</th>\n",
       "      <td>i find myself frustrated with christians becau...</td>\n",
       "      <td>love</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10714</th>\n",
       "      <td>i am one of those people who feels like going ...</td>\n",
       "      <td>joy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35177</th>\n",
       "      <td>i feel especially pleased about this as this h...</td>\n",
       "      <td>joy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122177</th>\n",
       "      <td>i was struggling with these awful feelings and...</td>\n",
       "      <td>joy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26723</th>\n",
       "      <td>i feel so enraged but helpless at the same time</td>\n",
       "      <td>anger</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     text emotions\n",
       "27383   i feel awful about it too because it s my job ...  sadness\n",
       "110083                              im alone i feel awful  sadness\n",
       "140764  ive probably mentioned this before but i reall...      joy\n",
       "100071           i was feeling a little low few days back  sadness\n",
       "2837    i beleive that i am much more sensitive to oth...     love\n",
       "18231   i find myself frustrated with christians becau...     love\n",
       "10714   i am one of those people who feels like going ...      joy\n",
       "35177   i feel especially pleased about this as this h...      joy\n",
       "122177  i was struggling with these awful feelings and...      joy\n",
       "26723     i feel so enraged but helpless at the same time    anger"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Preprocessing Data\n",
    "In the next steps we are going to create tokenize the text, create index mapping for words, and also construct a vocabulary. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenization and Sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retain only text that contain less that 70 tokens to avoid too much padding\n",
    "data[\"token_size\"] = data[\"text\"].apply(lambda x: len(x.split(' ')))\n",
    "data = data.loc[data['token_size'] < 70].copy()\n",
    "\n",
    "# sampling\n",
    "data = data.sample(n=50000);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Constructing Vocabulary and Index-Word Mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This class creates a word -> index mapping (e.g,. \"dad\" -> 5) and vice-versa \n",
    "# (e.g., 5 -> \"dad\") for the dataset\n",
    "class ConstructVocab():\n",
    "    def __init__(self, sentences):\n",
    "        self.sentences = sentences\n",
    "        self.word2idx = {}\n",
    "        self.idx2word = {}\n",
    "        self.vocab = set()\n",
    "        self.create_index()\n",
    "        \n",
    "    def create_index(self):\n",
    "        for s in self.sentences:\n",
    "            # update with individual tokens\n",
    "            self.vocab.update(s.split(' '))\n",
    "            \n",
    "        # sort the vocab\n",
    "        self.vocab = sorted(self.vocab)\n",
    "\n",
    "        # add a padding token with index 0\n",
    "        self.word2idx['<pad>'] = 0\n",
    "        \n",
    "        # word to index mapping\n",
    "        for index, word in enumerate(self.vocab):\n",
    "            self.word2idx[word] = index + 1 # +1 because of pad token\n",
    "        \n",
    "        # index to word mapping\n",
    "        for word, index in self.word2idx.items():\n",
    "            self.idx2word[index] = word  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a',\n",
       " 'aa',\n",
       " 'aaa',\n",
       " 'aaah',\n",
       " 'aabsolutely',\n",
       " 'aac',\n",
       " 'aahed',\n",
       " 'aaliyah',\n",
       " 'aaradhya',\n",
       " 'aaron']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# construct vocab and indexing\n",
    "inputs = ConstructVocab(data[\"text\"].values.tolist())\n",
    "\n",
    "# examples of what is in the vocab\n",
    "inputs.vocab[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Converting Data into Tensors \n",
    "For convenience we would like to convert the data into tensors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vectorize to tensor\n",
    "input_tensor = [[inputs.word2idx[s] for s in es.split(' ')]  for es in data[\"text\"].values.tolist()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[11609, 752, 8756, 1, 2350, 12147, 11882, 25071, 24557, 4212, 24322],\n",
       " [11609, 8747, 22258, 7581, 852, 22421, 26627, 11609, 26485, 12612]]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examples of what is in the input tensors\n",
    "input_tensor[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Padding data\n",
    "In order to train our recurrent neural network later on in the notebook, it is required padding to generate inputs of same length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def max_length(tensor):\n",
    "    return max(len(t) for t in tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "69\n"
     ]
    }
   ],
   "source": [
    "# calculate the max_length of input tensor\n",
    "max_length_inp = max_length(input_tensor)\n",
    "print(max_length_inp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_sequences(x, max_len):\n",
    "    padded = np.zeros((max_len), dtype=np.int64)\n",
    "    if len(x) > max_len: padded[:] = x[:max_len]\n",
    "    else: padded[:len(x)] = x\n",
    "    return padded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inplace padding\n",
    "input_tensor = [pad_sequences(x, max_length_inp) for x in input_tensor]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([11609,   752,  8756,     1,  2350, 12147, 11882, 25071, 24557,\n",
       "         4212, 24322,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0]),\n",
       " array([11609,  8747, 22258,  7581,   852, 22421, 26627, 11609, 26485,\n",
       "        12612,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "            0,     0,     0,     0,     0,     0])]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_tensor[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Binarization\n",
    "We would like to binarize our target so that we can obtain one-hot encodings as target values. These are easier and more efficient to work with and will be useful when training the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "### convert targets to one-hot encoding vectors\n",
    "emotions = list(set(data.emotions.unique()))\n",
    "num_emotions = len(emotions)\n",
    "# binarizer\n",
    "mlb = preprocessing.MultiLabelBinarizer()\n",
    "data_labels =  [set(emos) & set(emotions) for emos in data[['emotions']].values]\n",
    "bin_emotions = mlb.fit_transform(data_labels)\n",
    "target_tensor = np.array(bin_emotions.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 1, 0],\n",
       "       [0, 0, 1, 0, 0, 0]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_tensor[0:2] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>emotions</th>\n",
       "      <th>token_size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4298</th>\n",
       "      <td>i am feeling a bit inhibited in trying to clar...</td>\n",
       "      <td>sadness</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61926</th>\n",
       "      <td>i feel so elegant and sophisticated when i wea...</td>\n",
       "      <td>joy</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text emotions  token_size\n",
       "4298   i am feeling a bit inhibited in trying to clar...  sadness          11\n",
       "61926  i feel so elegant and sophisticated when i wea...      joy          10"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_emotion = lambda t: np.argmax(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_emotion(target_tensor[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "emotion_dict = {0: 'anger', 1: 'fear', 2: 'joy', 3: 'love', 4: 'sadness', 5: 'surprise'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sadness'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emotion_dict[get_emotion(target_tensor[0])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Split data\n",
    "We would like to split our data into a train and validation set. In addition, we also want a holdout dataset (test set) for evaluating the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40000, 40000, 5000, 5000, 5000, 5000)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating training and validation sets using an 80-20 split\n",
    "input_tensor_train, input_tensor_val, target_tensor_train, target_tensor_val = train_test_split(input_tensor, target_tensor, test_size=0.2)\n",
    "\n",
    "# Split the validataion further to obtain a holdout dataset (for testing) -- split 50:50\n",
    "input_tensor_val, input_tensor_test, target_tensor_val, target_tensor_test = train_test_split(input_tensor_val, target_tensor_val, test_size=0.5)\n",
    "\n",
    "# Show length\n",
    "len(input_tensor_train), len(target_tensor_train), len(input_tensor_val), len(target_tensor_val), len(input_tensor_test), len(target_tensor_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 Data Loader\n",
    "We can also laod the data into a data loader, which makes it easy to **manipulate the data**, **create batches**, and apply further **transformations**. In PyTorch we can use the `DataLoader` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_BUFFER_SIZE = len(input_tensor_train)\n",
    "VAL_BUFFER_SIZE = len(input_tensor_val)\n",
    "TEST_BUFFER_SIZE = len(input_tensor_test)\n",
    "BATCH_SIZE = 32 # great results at 64\n",
    "TRAIN_N_BATCH = TRAIN_BUFFER_SIZE // BATCH_SIZE\n",
    "VAL_N_BATCH = VAL_BUFFER_SIZE // BATCH_SIZE\n",
    "TEST_N_BATCH = TEST_BUFFER_SIZE // BATCH_SIZE\n",
    "\n",
    "embedding_dim = 256\n",
    "units = 100 # great results at 1024\n",
    "vocab_inp_size = len(inputs.word2idx)\n",
    "target_size = num_emotions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the data to tensors and pass to the Dataloader \n",
    "# to create an batch iterator\n",
    "\n",
    "class MyData(Dataset):\n",
    "    def __init__(self, X, y):\n",
    "        self.data = X\n",
    "        self.target = y\n",
    "        self.length = [ np.sum(1 - np.equal(x, 0)) for x in X]\n",
    "        \n",
    "    def __getitem__(self, index):\n",
    "        x = self.data[index]\n",
    "        y = self.target[index]\n",
    "        x_len = self.length[index]\n",
    "        return x, y, x_len\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = MyData(input_tensor_train, target_tensor_train)\n",
    "val_dataset = MyData(input_tensor_val, target_tensor_val)\n",
    "test_dataset = MyData(input_tensor_test, target_tensor_test)\n",
    "\n",
    "train_dataset = DataLoader(train_dataset, batch_size = BATCH_SIZE, \n",
    "                     drop_last=True,\n",
    "                     shuffle=True)\n",
    "val_dataset = DataLoader(val_dataset, batch_size = BATCH_SIZE, \n",
    "                     drop_last=True,\n",
    "                     shuffle=True)\n",
    "test_dataset = DataLoader(test_dataset, batch_size = BATCH_SIZE, \n",
    "                     drop_last=True,\n",
    "                     shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_dataset.batch_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Model\n",
    "After the data has been preprocessed, transformed and prepared it is now time to construct the model or the so-called computation graph that will be used to train our classification models. We are going to use a gated recurrent neural network (GRU), which is considered a more efficient version of a basic RNN. The figure below shows a high-level overview of the model details. \n",
    "\n",
    "![alt txt](img/gru-model.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Constructing the Model\n",
    "Below we construct our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmoGRU(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, hidden_units, batch_sz, output_size):\n",
    "        super(EmoGRU, self).__init__()\n",
    "        self.batch_sz = batch_sz\n",
    "        self.hidden_units = hidden_units\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.vocab_size = vocab_size\n",
    "        self.output_size = output_size\n",
    "        \n",
    "        # layers\n",
    "        self.embedding = nn.Embedding(self.vocab_size, self.embedding_dim)\n",
    "        self.dropout = nn.Dropout(p=0.5)\n",
    "        self.gru = nn.GRU(self.embedding_dim, self.hidden_units)\n",
    "        self.fc = nn.Linear(self.hidden_units, self.output_size)\n",
    "    \n",
    "    def initialize_hidden_state(self, device):\n",
    "        return torch.zeros((1, self.batch_sz, self.hidden_units)).to(device)\n",
    "    \n",
    "    def forward(self, x, lens, device):\n",
    "        x = self.embedding(x)\n",
    "        self.hidden = self.initialize_hidden_state(device)\n",
    "        output, self.hidden = self.gru(x, self.hidden) # max_len X batch_size X hidden_units\n",
    "        out = output[-1, :, :] \n",
    "        out = self.dropout(out)\n",
    "        out = self.fc(out)\n",
    "        return out, self.hidden  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Pretesting model\n",
    "Since eager execution is enabled we can print the output of the model by passing a sample of the dataset and making sure that the dimensions of the outputs are the expected ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "### sort batch function to be able to use with pad_packed_sequence\n",
    "def sort_batch(X, y, lengths):\n",
    "    lengths, indx = lengths.sort(dim=0, descending=True)\n",
    "    X = X[indx]\n",
    "    y = y[indx]\n",
    "    return X.transpose(0,1), y, lengths # transpose (batch x seq) to (seq x batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input size:  torch.Size([69, 32])\n",
      "torch.Size([32, 6])\n"
     ]
    }
   ],
   "source": [
    "# uncomment for cuda access\n",
    "# device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "device = torch.device(\"cpu\")\n",
    "\n",
    "model = EmoGRU(vocab_inp_size, embedding_dim, units, BATCH_SIZE, target_size)\n",
    "model.to(device)\n",
    "\n",
    "# obtain one sample from the data iterator\n",
    "it = iter(train_dataset)\n",
    "x, y, x_len = next(it)\n",
    "\n",
    "# sort the batch first to be able to use with pac_pack sequence\n",
    "xs, ys, lens = sort_batch(x, y, x_len)\n",
    "\n",
    "print(\"Input size: \", xs.size())\n",
    "\n",
    "output, _ = model(xs.to(device), lens, device)\n",
    "print(output.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Training the Model\n",
    "Now that we have tested the model, it is time to train it. We will define out optimization algorithm, learnin rate, and other necessary information to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Enabling cuda\n",
    "use_cuda = True if torch.cuda.is_available() else False\n",
    "# device = torch.device(\"cuda\" if use_cuda else \"cpu\") # uncomment for cuda\n",
    "device = torch.device(\"cpu\")\n",
    "\n",
    "model = EmoGRU(vocab_inp_size, embedding_dim, units, BATCH_SIZE, target_size)\n",
    "model.to(device)\n",
    "\n",
    "### loss criterion and optimizer for training\n",
    "criterion = nn.CrossEntropyLoss() # the same as log_softmax + NLLLoss\n",
    "optimizer = torch.optim.Adam(model.parameters())\n",
    "\n",
    "def loss_function(y, prediction):\n",
    "    \"\"\" CrossEntropyLoss expects outputs and class indices as target \"\"\"\n",
    "    # convert from one-hot encoding to class indices\n",
    "    target = torch.max(y, 1)[1]\n",
    "    loss = criterion(prediction, target) \n",
    "    return loss   #TODO: refer the parameter of these functions as the same\n",
    "    \n",
    "def accuracy(target, logit):\n",
    "    ''' Obtain accuracy for training round '''\n",
    "    target = torch.max(target, 1)[1] # convert from one-hot encoding to class indices\n",
    "    corrects = (torch.max(logit, 1)[1].data == target).sum()\n",
    "    accuracy = 100.0 * corrects / len(logit)\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 0 Val. Loss 0.2976\n",
      "Epoch 1 Batch 100 Val. Loss 0.2643\n",
      "Epoch 1 Batch 200 Val. Loss 0.2559\n",
      "Epoch 1 Batch 300 Val. Loss 0.2767\n",
      "Epoch 1 Batch 400 Val. Loss 0.2916\n",
      "Epoch 1 Batch 500 Val. Loss 0.2604\n",
      "Epoch 1 Batch 600 Val. Loss 0.2390\n",
      "Epoch 1 Batch 700 Val. Loss 0.1975\n",
      "Epoch 1 Batch 800 Val. Loss 0.2128\n",
      "Epoch 1 Batch 900 Val. Loss 0.1037\n",
      "Epoch 1 Batch 1000 Val. Loss 0.0787\n",
      "Epoch 1 Batch 1100 Val. Loss 0.1102\n",
      "Epoch 1 Batch 1200 Val. Loss 0.0677\n",
      "Epoch 1 Loss 0.1826 -- Train Acc. 58.0000 -- Val Acc. 87.0000\n",
      "Time taken for 1 epoch 143.81666803359985 sec\n",
      "\n",
      "Epoch 2 Batch 0 Val. Loss 0.0241\n",
      "Epoch 2 Batch 100 Val. Loss 0.0771\n",
      "Epoch 2 Batch 200 Val. Loss 0.0239\n",
      "Epoch 2 Batch 300 Val. Loss 0.0405\n",
      "Epoch 2 Batch 400 Val. Loss 0.0318\n",
      "Epoch 2 Batch 500 Val. Loss 0.0268\n",
      "Epoch 2 Batch 600 Val. Loss 0.0254\n",
      "Epoch 2 Batch 700 Val. Loss 0.0497\n",
      "Epoch 2 Batch 800 Val. Loss 0.0622\n",
      "Epoch 2 Batch 900 Val. Loss 0.0138\n",
      "Epoch 2 Batch 1000 Val. Loss 0.0345\n",
      "Epoch 2 Batch 1100 Val. Loss 0.0525\n",
      "Epoch 2 Batch 1200 Val. Loss 0.0226\n",
      "Epoch 2 Loss 0.0374 -- Train Acc. 91.0000 -- Val Acc. 91.0000\n",
      "Time taken for 1 epoch 168.25196933746338 sec\n",
      "\n",
      "Epoch 3 Batch 0 Val. Loss 0.0251\n",
      "Epoch 3 Batch 100 Val. Loss 0.0398\n",
      "Epoch 3 Batch 200 Val. Loss 0.0078\n",
      "Epoch 3 Batch 300 Val. Loss 0.0313\n",
      "Epoch 3 Batch 400 Val. Loss 0.0270\n",
      "Epoch 3 Batch 500 Val. Loss 0.0221\n",
      "Epoch 3 Batch 600 Val. Loss 0.0137\n",
      "Epoch 3 Batch 700 Val. Loss 0.0192\n",
      "Epoch 3 Batch 800 Val. Loss 0.0575\n",
      "Epoch 3 Batch 900 Val. Loss 0.0497\n",
      "Epoch 3 Batch 1000 Val. Loss 0.0351\n",
      "Epoch 3 Batch 1100 Val. Loss 0.0018\n",
      "Epoch 3 Batch 1200 Val. Loss 0.0076\n",
      "Epoch 3 Loss 0.0223 -- Train Acc. 93.0000 -- Val Acc. 92.0000\n",
      "Time taken for 1 epoch 168.72783184051514 sec\n",
      "\n",
      "Epoch 4 Batch 0 Val. Loss 0.0047\n",
      "Epoch 4 Batch 100 Val. Loss 0.0073\n",
      "Epoch 4 Batch 200 Val. Loss 0.0042\n",
      "Epoch 4 Batch 300 Val. Loss 0.0139\n",
      "Epoch 4 Batch 400 Val. Loss 0.0528\n",
      "Epoch 4 Batch 500 Val. Loss 0.0050\n",
      "Epoch 4 Batch 600 Val. Loss 0.0094\n",
      "Epoch 4 Batch 700 Val. Loss 0.0052\n",
      "Epoch 4 Batch 800 Val. Loss 0.0231\n",
      "Epoch 4 Batch 900 Val. Loss 0.0230\n",
      "Epoch 4 Batch 1000 Val. Loss 0.0416\n",
      "Epoch 4 Batch 1100 Val. Loss 0.0336\n",
      "Epoch 4 Batch 1200 Val. Loss 0.0127\n",
      "Epoch 4 Loss 0.0198 -- Train Acc. 94.0000 -- Val Acc. 91.0000\n",
      "Time taken for 1 epoch 168.09667444229126 sec\n",
      "\n",
      "Epoch 5 Batch 0 Val. Loss 0.0194\n",
      "Epoch 5 Batch 100 Val. Loss 0.0200\n",
      "Epoch 5 Batch 200 Val. Loss 0.0176\n",
      "Epoch 5 Batch 300 Val. Loss 0.0222\n",
      "Epoch 5 Batch 400 Val. Loss 0.0268\n",
      "Epoch 5 Batch 500 Val. Loss 0.0131\n",
      "Epoch 5 Batch 600 Val. Loss 0.0193\n",
      "Epoch 5 Batch 700 Val. Loss 0.0247\n",
      "Epoch 5 Batch 800 Val. Loss 0.0056\n",
      "Epoch 5 Batch 900 Val. Loss 0.0392\n",
      "Epoch 5 Batch 1000 Val. Loss 0.0249\n",
      "Epoch 5 Batch 1100 Val. Loss 0.0146\n",
      "Epoch 5 Batch 1200 Val. Loss 0.0025\n",
      "Epoch 5 Loss 0.0164 -- Train Acc. 95.0000 -- Val Acc. 91.0000\n",
      "Time taken for 1 epoch 169.05170464515686 sec\n",
      "\n",
      "Epoch 6 Batch 0 Val. Loss 0.0073\n",
      "Epoch 6 Batch 100 Val. Loss 0.0066\n",
      "Epoch 6 Batch 200 Val. Loss 0.0133\n",
      "Epoch 6 Batch 300 Val. Loss 0.0107\n",
      "Epoch 6 Batch 400 Val. Loss 0.0008\n",
      "Epoch 6 Batch 500 Val. Loss 0.0166\n",
      "Epoch 6 Batch 600 Val. Loss 0.0148\n",
      "Epoch 6 Batch 700 Val. Loss 0.0412\n",
      "Epoch 6 Batch 800 Val. Loss 0.0152\n",
      "Epoch 6 Batch 900 Val. Loss 0.0098\n",
      "Epoch 6 Batch 1000 Val. Loss 0.0029\n",
      "Epoch 6 Batch 1100 Val. Loss 0.0030\n",
      "Epoch 6 Batch 1200 Val. Loss 0.0359\n",
      "Epoch 6 Loss 0.0144 -- Train Acc. 95.0000 -- Val Acc. 91.0000\n",
      "Time taken for 1 epoch 169.29426193237305 sec\n",
      "\n",
      "Epoch 7 Batch 0 Val. Loss 0.0039\n",
      "Epoch 7 Batch 100 Val. Loss 0.0091\n",
      "Epoch 7 Batch 200 Val. Loss 0.0470\n",
      "Epoch 7 Batch 300 Val. Loss 0.0055\n",
      "Epoch 7 Batch 400 Val. Loss 0.0183\n",
      "Epoch 7 Batch 500 Val. Loss 0.0077\n",
      "Epoch 7 Batch 600 Val. Loss 0.0259\n",
      "Epoch 7 Batch 700 Val. Loss 0.0038\n",
      "Epoch 7 Batch 800 Val. Loss 0.0166\n",
      "Epoch 7 Batch 900 Val. Loss 0.0065\n",
      "Epoch 7 Batch 1000 Val. Loss 0.0211\n",
      "Epoch 7 Batch 1100 Val. Loss 0.0150\n",
      "Epoch 7 Batch 1200 Val. Loss 0.0244\n",
      "Epoch 7 Loss 0.0125 -- Train Acc. 96.0000 -- Val Acc. 91.0000\n",
      "Time taken for 1 epoch 169.72389197349548 sec\n",
      "\n",
      "Epoch 8 Batch 0 Val. Loss 0.0166\n",
      "Epoch 8 Batch 100 Val. Loss 0.0117\n",
      "Epoch 8 Batch 200 Val. Loss 0.0112\n",
      "Epoch 8 Batch 300 Val. Loss 0.0110\n",
      "Epoch 8 Batch 400 Val. Loss 0.0038\n",
      "Epoch 8 Batch 500 Val. Loss 0.0538\n",
      "Epoch 8 Batch 600 Val. Loss 0.0105\n",
      "Epoch 8 Batch 700 Val. Loss 0.0020\n",
      "Epoch 8 Batch 800 Val. Loss 0.0206\n",
      "Epoch 8 Batch 900 Val. Loss 0.0013\n",
      "Epoch 8 Batch 1000 Val. Loss 0.0067\n",
      "Epoch 8 Batch 1100 Val. Loss 0.0172\n",
      "Epoch 8 Batch 1200 Val. Loss 0.0070\n",
      "Epoch 8 Loss 0.0114 -- Train Acc. 96.0000 -- Val Acc. 91.0000\n",
      "Time taken for 1 epoch 170.62678408622742 sec\n",
      "\n",
      "Epoch 9 Batch 0 Val. Loss 0.0036\n",
      "Epoch 9 Batch 100 Val. Loss 0.0088\n",
      "Epoch 9 Batch 200 Val. Loss 0.0289\n",
      "Epoch 9 Batch 300 Val. Loss 0.0084\n",
      "Epoch 9 Batch 400 Val. Loss 0.0118\n",
      "Epoch 9 Batch 500 Val. Loss 0.0006\n",
      "Epoch 9 Batch 600 Val. Loss 0.0015\n",
      "Epoch 9 Batch 700 Val. Loss 0.0009\n",
      "Epoch 9 Batch 800 Val. Loss 0.0303\n",
      "Epoch 9 Batch 900 Val. Loss 0.0105\n",
      "Epoch 9 Batch 1000 Val. Loss 0.0049\n",
      "Epoch 9 Batch 1100 Val. Loss 0.0012\n",
      "Epoch 9 Batch 1200 Val. Loss 0.0044\n",
      "Epoch 9 Loss 0.0097 -- Train Acc. 97.0000 -- Val Acc. 90.0000\n",
      "Time taken for 1 epoch 168.92880845069885 sec\n",
      "\n",
      "Epoch 10 Batch 0 Val. Loss 0.0003\n",
      "Epoch 10 Batch 100 Val. Loss 0.0002\n",
      "Epoch 10 Batch 200 Val. Loss 0.0084\n",
      "Epoch 10 Batch 300 Val. Loss 0.0059\n",
      "Epoch 10 Batch 400 Val. Loss 0.0118\n",
      "Epoch 10 Batch 500 Val. Loss 0.0134\n",
      "Epoch 10 Batch 600 Val. Loss 0.0022\n",
      "Epoch 10 Batch 700 Val. Loss 0.0257\n",
      "Epoch 10 Batch 800 Val. Loss 0.0060\n",
      "Epoch 10 Batch 900 Val. Loss 0.0158\n",
      "Epoch 10 Batch 1000 Val. Loss 0.0012\n",
      "Epoch 10 Batch 1100 Val. Loss 0.0146\n",
      "Epoch 10 Batch 1200 Val. Loss 0.0030\n",
      "Epoch 10 Loss 0.0085 -- Train Acc. 97.0000 -- Val Acc. 91.0000\n",
      "Time taken for 1 epoch 170.37976574897766 sec\n",
      "\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 10\n",
    "\n",
    "for epoch in range(EPOCHS):\n",
    "    start = time.time()\n",
    "    \n",
    "    ### Initialize hidden state\n",
    "    # TODO: do initialization here.\n",
    "    total_loss = 0\n",
    "    train_accuracy, val_accuracy = 0, 0\n",
    "    \n",
    "    ### Training\n",
    "    for (batch, (inp, targ, lens)) in enumerate(train_dataset):\n",
    "        loss = 0\n",
    "        predictions, _ = model(inp.permute(1 ,0).to(device), lens, device) # TODO:don't need _   \n",
    "              \n",
    "        loss += loss_function(targ.to(device), predictions)\n",
    "        batch_loss = (loss / int(targ.shape[1]))        \n",
    "        total_loss += batch_loss\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        batch_accuracy = accuracy(targ.to(device), predictions)\n",
    "        train_accuracy += batch_accuracy\n",
    "        \n",
    "        if batch % 100 == 0:\n",
    "            print('Epoch {} Batch {} Val. Loss {:.4f}'.format(epoch + 1,\n",
    "                                                         batch,\n",
    "                                                         batch_loss.cpu().detach().numpy()))\n",
    "            \n",
    "    ### Validating\n",
    "    for (batch, (inp, targ, lens)) in enumerate(val_dataset):        \n",
    "        predictions,_ = model(inp.permute(1, 0).to(device), lens, device)        \n",
    "        batch_accuracy = accuracy(targ.to(device), predictions)\n",
    "        val_accuracy += batch_accuracy\n",
    "    \n",
    "    print('Epoch {} Loss {:.4f} -- Train Acc. {:.4f} -- Val Acc. {:.4f}'.format(epoch + 1, \n",
    "                                                             total_loss / TRAIN_N_BATCH, \n",
    "                                                             train_accuracy / TRAIN_N_BATCH,\n",
    "                                                             val_accuracy / VAL_N_BATCH))\n",
    "    print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Module.parameters of EmoGRU(\n",
       "  (embedding): Embedding(27348, 256)\n",
       "  (dropout): Dropout(p=0.5)\n",
       "  (gru): GRU(256, 100)\n",
       "  (fc): Linear(in_features=100, out_features=6, bias=True)\n",
       ")>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Evaluation on the Testing Data\n",
    "Now we will evaluate the model with the holdout dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy:  92.08974358974359\n"
     ]
    }
   ],
   "source": [
    "test_accuracy = 0\n",
    "all_predictions = []\n",
    "x_raw = []\n",
    "y_raw = []\n",
    "\n",
    "device = \"cpu\" # we don't need GPU to do testing\n",
    "model.to(\"cpu\")\n",
    "\n",
    "for (batch, (inp, targ, lens)) in enumerate(test_dataset):          \n",
    "    predictions,_ = model(inp.permute(1, 0), lens, device)        \n",
    "    batch_accuracy = accuracy(targ, predictions)\n",
    "    test_accuracy += batch_accuracy\n",
    "    \n",
    "    x_raw = x_raw + [x for x in inp]\n",
    "    y_raw = y_raw + [y for y in targ]\n",
    "    \n",
    "    all_predictions.append(predictions)\n",
    "    \n",
    "print(\"Test Accuracy: \", test_accuracy.detach().numpy() / TEST_N_BATCH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Confusion Matrix\n",
    "The test accuracy alone is not an interesting performance metric in this case. Let's plot a confusion matrix to get a drilled down view of how the model is performing with regards to each emotion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default Classification report\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "       anger       0.94      0.93      0.93       668\n",
      "        fear       0.90      0.85      0.88       572\n",
      "         joy       0.94      0.95      0.94      1681\n",
      "        love       0.89      0.80      0.84       433\n",
      "     sadness       0.96      0.97      0.97      1462\n",
      "    surprise       0.72      0.83      0.77       176\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      4992\n",
      "   macro avg       0.89      0.89      0.89      4992\n",
      "weighted avg       0.93      0.93      0.93      4992\n",
      "\n",
      "\n",
      "Accuracy:\n",
      "0.9268830128205128\n",
      "Correct Predictions:  4627\n",
      "precision: 0.89\n",
      "recall: 0.89\n",
      "f1: 0.89\n",
      "\n",
      "confusion matrix\n",
      " [[ 620   15    9    0   24    0]\n",
      " [  24  489    1    0   17   41]\n",
      " [   5    6 1603   44   12   11]\n",
      " [   0    2   80  345    4    2]\n",
      " [  13   13   10    0 1424    2]\n",
      " [   0   18   11    0    1  146]]\n",
      "(row=expected, col=predicted)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAGGCAYAAABPDDfEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzsnXd8FMX7x99PwFAChBKSEAQpQaQlQAKBoJTQQUClI1IFRHpT6c0uSm+CispXpSMiitJFQhKqP1EpCoJiGiLRBAiQ+f2xd+EuuQu5IyQXMm9e+yI3+8zOZ2f39rmZeWZWlFJoNBqNRnOvcctpARqNRqPJG2iHo9FoNJpsQTscjUaj0WQL2uFoNBqNJlvQDkej0Wg02YJ2OBqNRqPJFrTD0Wg0Gk22oB2ORqPRaLIF7XA0Go1Gky1oh6PRaDSabEE7HI1Go9FkC/lzWoBGo9HkdUSkIOB+F4dIVkpdyyo99wrRi3dqNBpNziEiBclf+Co3k+7mMNFARVd3OrqFo9FoNDmLOzeTKFCjP+RzopFzK5nrJz7wxWghaYej0Wg0mjuQzx1xwuHkpj4q7XA0Go3GFRBAxLl8uQTtcDQajcYVEDdjcyZfLkE7HI1Go3EFRJxs4eSeJk7ucY0ajUajydXoFo5Go9G4ArpLTaPRaDTZQh7oUtMOR6PRaFwCJ1s4uWhkRDscjUajcQXyQAsn97hGjUaj0eRqdAtHo9FoXAEdNKDRaDSabCEPdKlph6PRaDSuQB5o4eQepRqNRqPJ1egWjkaj0bgCuktNo9FoNNlCHuhS0w5Ho9FoXAERJx1O7mnh5B7XqNFoNJpcjW7haDQajSvgJsbmTL5cgnY4Go1G4wroMRyNRqPRZAt5IEot97hGjUaj0eRqtMPRACAiVUTkGxG5IiJKRJ7I4uNXMB23X1Ye935ARM6JyKqc1pEbEJGmpvuoqUXaKhE5l806sv5+NnepObPlEnKP0jyAiFQWkeUi8puIXBORBBH5XkRGiUihe1z8h0AtYDLwDHDoHpd33yEi1UVkhohUyEEN5gehEpHONvbPMO3zygl9mgwwd6k5s+US9BiOiyAi7YF1wHXgI+BHwB14FHgLqAEMvkdlFwIaAq8opRbdizKA34FCwI17dHxXoDowHdgDnHMgX1Ug5R7omSYiG5VS6h4c25UYxP3w41kHDWiyAxGpCHyG8VAOU0r9ZbF7sYj4A+3voYTSpv//uVcFmB561+7V8XMbIiJAQaXUVaXU9XtQxDGgNvAksPEeHB8AEfFQSiXeq+NnBqXU/fEjRgcNaLKJF4AiwMA0zgYApdQZpdR882cRyS8iU0XkVxG5bhoDeFVECljmM6VvFZFHRSTS1E33m4j0sbCZgeHoAN4ydbecM+2z2Tdu7pZJk9ZSRPaLyD8i8p+InBSRVy322+zzFpEwEflORBJNeT8XkWq2yhMRf5Omf0xjTR+ISOGMqxZEZI+I/CgiASKyV0SSROSMiHQx7W8iIhEictWku0Wa/A+JyBLTvqsicklE1ll2nZnOa53p426Lbq2maa5FaxE5BFwFhljsW2X6W0Rkt4jEiYi3xfHdReT/TNfc407njPED5hRGK+eOTyQR6Soih03nFy8iq0WkbBqbVaZrW1lEtonIv8D/squOM9BudZ+atCg7Wz8Lu+IiMk9ELpi+R2dE5EUR6yaDyW6V6Z77R0Q+BIrfSZcmPdrhuAYdgN+UUgcyab8SmAUcAcYAe4GJGA+ZtPgD64FvgXHAZWCViNQw7d9oOgbApxjjN6MdEW861lagADDNVM4WoNEd8rUAtgPewAzgHSAU+N7Og2YtUBTjXNcC/TC6sDJDCZPGCAwHfx34TES6Y9TbNuAlwANYLyJFLfLWM+n6DBgJLAOaA3ssHN4+YIHp71cx6vEZ4GeL41TFqONvgVEYrRArTC3BAUBBUzlmZmJ0q/bPZIviFvAyEIjRyrGL6SG81pRnIrACeArYLyJpH6z5Ma5ZLDAe2GCx717XcWZ5hdv1b962m/bFms65MMb3pjdGF/ZI4HvgNYz70Fw3AnxuOsZqYArwIMaYZ9aSB4IGdJdaDiMixYCyGDd1ZuwDgb7ASqXUIFPyEhGJBcaLSDOl1G6LLFWBxkqp70z51wIXgP7AeKXUDyKSAMwFjiilVjtxGi0xxpvaKqXiHcj3FvA30FAp9bdJ32bgKMYDtm8a+6NKqYHmDyJSChgIvJiJsvyAXkqpT015vwV+AT4BQpVSEab0nzEeTp2BVaa8Xyql1lseTES+AMJNdh8rpX4Tke8wHlzfKqX22NDgD7RRSm23sS8VpdRZERkHLBeRp4EzwARgvlJqXybO1cwnwFSMVs4mW2M5IvIA8AbGmGFjpdQ1U/p+DOcxBmunXgBYp5SaaKO8e1rHmT1ppdS3aY4TCoQB7yultpmSxwKVgTpKqdOmtOUichGYICJvK6UuAB2BxsALSqm3TMdbClh+x7IG3aWmyQaKmf7/N5P27Uz/v5Mm/W3T/2nHen4yOxsApVQccBKo5IjIO2Ae++mUtjvCHiJSBmOMYZXZ2Zj0/YDRAmhnI9uyNJ+/A0qZnPad+A+LFqBS6qRJ98/mB6EJ89+VLGyvWuh+wOTozpjy181E2WbO3snZWJT5LsZDeSHGw/ZXYJIDZaGUsmzl2AtzD8ZoYS4xOxtT3i8xnIWtscOldo7lCnVshYj4YrTwjwHPW+zqinH/XBYRL/MG7ADyYTgZMO7Dm1ics6leFzqryT7Otm5yz2M89yi9f0kw/V80Q6vbPIQR0XTGMlEpFY3x5Xwojf15G8e4jNH9kVWsweiOWAnEiMhnItLtDs7HrPOkjX0/A142xirSnstl0/+ZOZc/bPzCv4LR2ktFKXUl7TFFpJCIzBKRCxjdRPFAHEY/vmcmyjZz1gFbMFpvhYEqQD/Lh7ID/A/jXrE3lpPRdfiF9PfTTeAPO2W5Qh2nIiL5MboK8wFPpQnOqAK0MZVhue0w7TePnz0E/KWU+i/N4W3Vl+YO6C61HEYplWBqxtd0NGsm7W7ZSc9MO9xeGfmsjJS6KiKNgWYYv4jbAN2BXSLSyvSLMCu4m3Oxlzczx1yI0QU5D6OL5wpG3XyGYz/aHHUYTTG6sMCYIxXuYH6UUrdE5GWMrqtOjua3wXWllL0QbleoY0vewgj3b6GUSusk3TBa0m/ayXvKyTKdJw90qWmH4xpsBQaLSEOl1J0eKr9jfFmqYDEgLSI+GL8Gf7eTzxkuYzsaJ+2vXkwPoZ2mbayITMIYvG3G7V+Nlph1VrWx7xEgPqfDbS3oAnyolBpnThCRgqSvmyyb72LqclwIfAMkA3NEZLtSypnrax7sno4RzGGJ5XXYlWZfVbL2fsqIzNZxphCRHhjBL6OVUnttmPwKFFFK2bo3LfkdaC4iRdK0cmzdt3eHfh+OJpt4E0gEVpochxWmMNRRpo/mQc+0kWRjTf9/mYW6fgU8RSTAQksZ0kQ9iUhJG3nNEVgFbOzDFP59DOhrGQklIjWBVtw+T1fgFulbUSNI09LDuIaQNSGzKzC+nwMxJvzeBN6z0y2WIRZjObUxBsEtOYQRufWcWITVi0hboBpZez9lRGbr+I6Y7qGVwGrL6QRpWAs0FJHWNvIXN3XHgXEf5geGWuzPZ9KWtegoNU12oJT6VUR6YYyF/CwilisNhGIMcK4y2R43zQMYbHpQ7wXqY0R0bU4ToXa3fIYRwbRJRBZgjCcMxehusBzInWbqUvsS4xehN8YA7R/A/gyOPwH4CggXkfcwViIYgdGdMiMLz+Nu2Qo8IyJXgJ8wddMAl9LYHcN4cL4oIp4YYxG7lFKxjhQmIv0xuib7mbuCRGQERktlKLDEiXP4H0bEWm3LRKXUDRF5EfgA2CsinwI+GGHb5zCiF7ODzNZxZvjA9P8+EemdZt8BpdRvGN1tHYGtYsyBOowRrl0Lo7VVAWMc6QuM8cnXTaH6P2GEjDs1rpTX0Q7HRVBKbTG1JCZg9LUPxXhg/YAxr2WFhfmzwG8Y81CeBKIx5g/MzGJNl0TkSYyIuDcxBr0nYnTnWTqcLRhf0AGAF8YXdS8w3WKA2Nbxd4hIG5PuWRjL3uwFXlRKOTrAfi8ZheFInsaYH/M9xsPQKuJMKRUtIs9h1NF7GL/Om2Ga+5EZRORBjIf8F0qp1LkeSqn/ibE22psi8pWj9aOUumkay/nAxr5VIpKEMUfmDYyW2iaM63DPVp9IQ6bqOJOUxnAe79rY1x9jzluSiDTBiPzrCvTBCOA5hdH1eAWMrmIR6YgxttQbo9t0C8Z38qgT2uyTB8Zw5P5fZkmj0WhcF1NY/5UCbeciDzi+Rq+6cZXrX40B8FRKJdzJPifRLRyNRqNxBfJAC0c7HI1Go3EF8sBq0blHqUaj0WhyNbqFo9FoNK6A7lLTaDQaTXYgIjgxzUo7HM2dMU3g8yPzi3ZqNBrXpShw8W7erqodjuZe4of9RRA1Gk3u40Hgz5wW4cpoh5Nz/AvgHjwKyW9z9ReX4Lcttl574jo49Yswm8nn5voar93IqvVV7w0FH3B4hZts49+EBPwrloO77a0QMrcMra18uQTtcHIYyV/ApR1OsWKZedVMzqEdTtbgrh1OjqO71DQajUaTLeQFh6Pn4Wg0Go0mW9AtHI1Go3EB8kILRzscjUajcQG0w9FoNBpN9pAHotT0GI5Go9FosgXdwtFoNBoXQHepaTQajSZbMNbudMbhZL2We4XuUnNhhjxZn1/WjuHyjqnsWz6Y4Gpl7drmz+fGxH5NOfHZaC7vmErEB8/Tsr6/lc2gJ+oRuep5Yr6eRMzXk9izdBCtQqrclcZ3ly2hxsOV8PIsTLPHGnIoKjJD+00b1lE3oDpenoUJCQpk+9fbUvfduHGDqZNfIiQoEJ+SRalS8UEGD+jLXxcvOq1v+dLFVH+4IqWKFaLpow3uqG/jhnXUqVWNUsUKUb9uANu/2ma1//PNG+nYrjXly3hRpIAbPxw/5rQ2M8uWLKaqfwWKFynIY6EhREVmrHHD+nUE1nyE4kUKEly7Fl+n0aiUYtaMaVQsV4YSRQvRrnULzpw+7bS+FcuWEPBIZXxLeNCicUMO36EON29cT/3aNfAt4UFovdp88/U2u7ZjRjxPicL5WbpovtP6wPXrMDMIktrKcWhz0uOIyDAROSci10QkQkTq38F+tIicFJGrInJBROaKSEFHytQOx0XpElaTN4a34ZVVe2j47DJ+OBPNlrf7ULq4h037GYOa82zHYMbO+5I6zyxi5edRrHm1J4FVfFNt/oxNYOqybwl9dhmNBi1nz5HfWPdaT6pVKO2Uxg3r1jDxhXG8NHkq+w8eomatAJ7s0Ja42Fib9gfDD9C/z9P06TeA/RGHebxDJ3p2fYqfTvwIQFJSEsePHuHFiZP57uAh/vfZek6fPkX3Lk84pW+9Sd/EydPYH3GYmrUCeOLxNsRmpO+ZXvTtN4DvI47weMdO9Oj6JCdM+gCSEhNp2KgRs1553SlNaVm3dg0vThjL5CnTCY88QkBAIB3bt7arMfzAAfr27knf/gM5GHWUDp2eoFvnJzjx422Nb895kyWLFrBg8TL2fR+Bh4cHHdq35tq1aw7r27h+LVNeGs+Lk6ay50AUNWsF0rlTO7vXOOLgAZ7t+zS9+/Znb/gh2j/ekd7dO6deY0u2fr6ZQ5ERlCnj57AuS1y9DjOLU87GyW44EekOvAPMBOoCx4HtIuJtx74X8LrJvhowEOgOvOpQuXexuKnmLkh9j3mDF2wubbNv+WAO//wnY+Z9abbnzIZxLN0QwZz/fZfO/rdN43njo30s33T7l92ns7tzNfkmA2ZvsKvjzy9fYtKSb/jwyyM298ftmGE3b7PHGlI3KJi35y0EICUlhUf8H2LI0OGMm/BiOvu+vXuQmJjI+k1f3D5G41ACAgKZv2ipzTIOH4qi6aMN+OnUWcqVL59uf0ZftqaPNqBuUDDvzF+Uqq9q5fI89/xwxk14KZ19n6d7kJSYyPrNFvoea0itgEAWLF5mZfv7uXPUqFqJA5FHCAisbVcDZLy0zWOhIQQF12Pegtsa/SuWY+iwEUx4Ib3G3r26k5SYyMbPt6amNW7UgMDA2ixcsgylFJXK+zFyzDjGjB0PwJUrV3iorA/vvreKbt172NRhby21Fo0bUieoHm/NXZCqr2aVCgwaOowx49Nf4wHP9CQxMZE1G7ekprVsEkrNgNrMXbgkNe3in3/Sskko67dso/tTHRk6fCRDh4+yW08ZLW2T03WYkJCATylPAE+lVIJdoXYwPwtKdF+JuBd2NDsqOYnLa54FY/FQy/XcriulrtspMwKIUkoNN312Ay4AC5VS6X5NicgioJpSqrlF2ttAiFLq0cxq1S0cF+SB/Pmo83AZdh3+NTVNKcWuQ79Sv8aDNvO4P5Cfa8k3rdKuJt8ktFb6hzSAm5vQtXlNPAq6E3HigsMak5OTOXrkME3DUu8/3NzcaNqsOZER4TbzRB48SLOwFlZpLVq0IjLioN1yEq5cQUTwLF7cKX2W5bm5udEsrAWRB22XFxkRTjOL8wFo3jJjfXeDWWNYc2uNYWEtiDxouw4jDoanq8OWrVoTYbI/d/Ys0dHRhFnYeHp6Uq9+SKqNI/qOHT1C02bW17hJWHOi7NRJZMRBq3sCIKxFK6Iib9unpKTw3LN9GTFmHNWq13BIky2NrlyHDiF3sRn8AVyx2GyuvCsi7kAQsMOcppRKMX1uaEfdASDI3O0mIpWAdoD9/lIb6KABF8TLszD58+cj9u9Eq/TYy4lUfch299eOyDOM7B7K/uPn+O3PyzQLqkSnxtXI52b9m6JGJW/2LB1EQff8/Hc1me6TP+WXc3EOa7wUH8+tW7fw9vaxSvf28eH0qZM288TEROPt7Z3OPiYm2qb9tWvXmDZlIl279XB4EdFUfT5p9Hl7c+rkL7b1RUdTOp29fX13S3wGdXgyA43pz+m2xujo6NRjpD2mo+dhrsPSPtbXrLS3N6ft6IuNiaZ0mvMp7e1DrEXZ895+k/z58zPk+REO6bGFq9ehQzjZPaZu50nXwrGTxQvIB8SkSY8BHrFZhlKfiIgXsN/0Lq/8wDKllENdatrh3CeMX7CNJS904vjqkSil+O3iZT7adpS+7eta2Z06f4mQAUvx9CjAk81qsGLyU7Qa8b5TTudecuPGDfo83R2llFVXjCZ3c+zIYZYvXsieA1G5YqXv7OQuxmPMf/7rTJdeJstoCkwCngciAH9gvohMVUrNzuxxdJeaCxJ/JYmbN2/hXdI6QMC7hAfRl2y/ciP+nyS6TfqUUq1epmrXdwh8egGJV5M5e/Gyld2Nm7f47c+/OXrqL6Yt38H/nYlmWJcGDmss5eVFvnz5iI21/pEUGxOT7pehGR8f33QDubExMfj4+FqlmZ3NhfPn+fzL7U69IiFVX0wafbGx6cpL1efrS1w6+/T6sgqvDOrQ19e+xvTndFujOV86Gxv1fCfMdRgXY33N4mJj8bZzLG8fX+LSnE9cbEyqffiB/cTFxVKrakW8ihbAq2gBLpz/nSkvTSDgkcoO6QPXr0MXJR64BaT9ovoA9ppws4GPlVIrlVL/p5TahOGAJprGfzKFdjj3ABF54G7y37h5i6On/qJZUCXLY9IsqBKRJzJ+Sej15JtcjP+X/PnceKJJdbbut92tYMZNhALujjd03d3dqVM3iL27d6WmpaSksHfPLuqH2O4Grt+gAXt277RK27VrB/VDbjs8s7P59cwZtmz7hlKlSjmszVKfZXkpKSns2b2T+g1sO9j6IQ3ZY3E+ALt3WuvLSswad++y1rh7907qN7BdhyENGqarw507viXEZF+hYkV8fX3ZbWGTkJBAVGREqo0j+mrXqcvePdbXeN/uXdSzUyf1QxpY3RMAu3ftoF59w757z97sjzzKvoOHU7cyZfwYMWYcG7Y4NByQqtGV69ARsitKTSmVDBwGLAMA3Eyf7Q1SFQZS0qSZI00yLSBXd6mJSBtgClAT4+TDgVFKqV9FpAJwFugMjABCgNPAc0qpcItjDAKmAaWA7cB3wDSlVHELm07AdKA6cBH4EHhFKXXTtF9hNDXbYly0t4AZabQWACzD0YpmdG4L1hxgxaQnOfzLRQ79/AfDuzakcCF3PtpmRJOtnPwUF+MTmLbcGPerV/1B/LyKcvx0NGVLF2PygGa4uQnvfLI/9ZizhrRg+8HTXIi5QtHC7nRvGUDjOhXoMO7jjKTYZfjI0Qx5tj916gYRVK8+SxbOJykxkWf69ANg8IC+lPEry8yXjW7eocNG0rZlMxbMe4fWbduxYe0ajh4+xEJTBNiNGzfo3bMrx48eZd2mLaTcukWMqT+9RMmSuLu7O6Zv1BiGDOxH3aBggoLrs3jhPJISE+ndpz8Agwb0xc/Pj5kvvwbA88NH0qZFUxbMfZvWbduzft1nHDl8iAVLlqce8++//+aPC+dT5wadMo1X+fj44mPnF3VGjBw9lkED+hIUFExwvfosWmBo7NPX0DiwXx/8ypZl9iuGxmHDR9GqeRPmzX2btm3bs26toXHx0ncB46E1bORo3nj1Zfz9q1ChQkVmzphKGT8/OnZyPLz8+ZFjeH6QcY3rBtdj6aIFJCYl8vQz/QB47tl+lPHzY/os4xoPGTaCx1uFsWj+O7Rq046N69Zw7Mhh5i0yrnHJUqUomeZHRP4HHsDHx5cqD1d1WF9uqMNMk71rqb0DfCgih4BIYDTgAXwAICIfAX8qpcyBB18AY0XkKLe71GYDXyilMv32vlztcDAq6B3gB6AIMAvYJCKWcaqvAOMxnM0rwKci4q+UuikijYBlwIvAFqAFRiWmIiKPAR8BIzGcUWXgXdPumRamM4CXMC6cdbiYwUQMp5Up1u/6Ea/ihZk2MAyfkkX44Uw0ncZ/TOxlI5CgnI8nKRYh7QXc8zN9UHMqlinBf1eT2X7wNANnb+DKf7fnDZQu7sF7k5/Ct1RRriRe48dfY+gw7mN2Hfo1XfmZoXPX7sTHx/PKrBnExEQTEFibjVu2pXapXbhwAbEIWmjQMJT3P1zNrBnTmDltMpX9q/Dpuo1Ur1ETMEJlt201QpJD61uPPW3bvpPHmjR1SF+Xrt2Jj4vj5VnTiYk29G364it8UvWdxy2tvo/+x+zpU5lh0vfZuk3UMOkD2LZ1C88NGpD6uV/vngBMnDKNyVNnOKQPoGs3Q+OsmdNSNX6+9Wu7GhuGhrLq40+YOX0K06dMwr9KFdZu2EyNmrc1jhv/AkmJiQwfOph//vmH0EaPsmXr1xQs6NAcPQCe6tKN+Lg4Xp09g9iYaGoFBLJ+85ep1/iPNPpCGoSyYtVqXpk5jdnTp1DJvwqr12xIvcb3Alevw8ySBWM4mUYptUZESmM8M32BY0AbpZS5H7E81i2alwFl+r8sEIfhhCY7pPV+modjiqKIA2oB/2G0cJ5VSr1n2l8dOIERT/6LiHwGFFFKPW5xjNXA4+YWjojsAHYqpV6zsOkNvKmU8jN9VsA8pdSYDLTZauH8YW8ejquQ0TwcVyA3DDznhldM25uH4yq48iums2oejtczq3BzYh5OSnIS8R/3c7r87CRXj+GISBUR+VREfhORBOCcaZfl5JMfLP7+y/S/Oc6zKkZz0pK0nwOBaSLyn3kDVgBlRMTy7jiUkVal1HWlVIJ5wzp8UaPR5HGyc6WBnCK3d6l9AfwODMIYW3EDfgQsO/tvWPxtbs454miLYHSFbbSxz3Kdi0Qb+zUajSZTZGeXWk6Rax2OiJTCaKEMUkp9Z0rL9BILJk4C9dKkpf18BKiqlDrjlFCNRqPJBNrhuDaXgUvAYBH5C6MbzdEVFRcC+0RkLEZrKQwj0sxyYGsWsFVEzgPrMQbSAoGaSqkpd3cKGo1GY0K/8dN1Ma390wNjTaAfgbnABAeP8T3wHDAWY7XUNqbjXLOw2Q48DrQCooCDwBiMrjyNRqPRZJLc3MJBKbUDY26MJWLnb5RS/9hIW4ERBGBkEFkBnEljsx1jjo49HbnoN4ZGo3FFdJdaHkBExgPfYgz6twX6Ykzi1Gg0mmxDO5y8QX3gBYx5Mb8BI5VSK3NWkkajyWtoh5MHUEp1y2kNGo1GkxfI8w5Ho9FoXII8EKWmHY5Go9G4ALpLTaPRaDTZQl5wOLl2Ho5Go9Foche6haPRaDQugOBkCycXDeJoh6PRaDQuQF7oUtMOR6PRaFyBPBClpsdwNBqNRpMt6BaORqPRuAC6S01zzzm56SWKFSuW0zLsUrrHezktIUMurxuU0xLuC9xc/KF1JenGnY1yiH+zSJt2OBqNRqPJFkSMzZl8uQXtcDQajcYFMByOMy2ceyDmHqGDBjQajUaTLegWjkaj0bgCTnap5aawaO1wNBqNxgXQQQMajUajyRbyQtCAHsPRaDQaTbagWzgajUbjAri5CW5ujjdXlBN5cgrtcDQajcYFyAtdatrhaDQajQuQF4IG9BiORqPRaLIF7XBcmJXLlxBYrTJlSnrQoklDDh+KzNB+88b1hNSpQZmSHjSqV5tvv95m13bsyOcp6ZGfpYvm35XGIW2r88vyHlxe0599b3QiuErpDO2HP16T44u68vdn/Tm9oidv9m9AgQfype6f3L0uVzcNstqOLezqtL5lSxZT1b8CxYsU5LHQEKIiM67DDevXEVjzEYoXKUhw7Vp8/ZV1HSqlmDVjGhXLlaFE0UK0a92CM6dPO60vN2hcsWwJtapWwrt4YcIea8jhqIz1bdqwjuDA6ngXL0zD4EC+SXMfvvbyTIIDq1OmVFHKlylFx3atOBQZ4bQ+gPdXLCW4VhUe8i5K27BGHDkclaH9lk3reTS4Jg95F6Vpwzrs+OYrq/2+nu42t8Xz374rnRlh7lJzZsstaIfjomxcv5YpL43nhYlT2f19FDVrBdKlUzviYmNt2kccPMCgfk/zdJ/+7DlwiHYdOtK7R2d+OvFjOtutWzZzKDKCMmX87kpjl0aVeKN/A15Zc4SG4zbxw7lLbJnWltKeBW3ad3+sMrOfqcera45Qe8Q6nlu0jy6PVmJW73pWdifO/02F/qtTt+aTtjilb93aNbw4YSyTp0wnPPIIAQGBdGySNMXIAAAgAElEQVTfmlg7dRh+4AB9e/ekb/+BHIw6SodOT9Ct8xOc+PF2Hb49502WLFrAgsXL2Pd9BB4eHnRo35pr167dlxo3rFvDpBfH8eLkqewLP0TNgACe7NjW/n0YfoCBfZ/mmb4D+O7gYdp36ESvbk9Z3Yf+/lV4a+4CDhw6zvad+yj/0EM82aEN8XFxDusD2LxhLTMmTWDci1P4Zl8ENWoG0PPJ9sTF2dYYFRHO0IHP0POZ/nz7XSRt23ekf68u/PzTbY0/nDpvtc1dvAIR4fGOTzqlMTOYu9Sc2XILopTKaQ15EhEpBlw599ffNleLbtGkIXWD6vHmOwsASElJodbDFRj03DBGj38xnf2APj1JSkzksw23H84tm4ZSK6A27yxYkpp28eKftGwSyvrPt9Gjc0eeGzaSocNH2dXp9/T7dvfte6MTh8/EMWbFAdM5wZkVvVi67QRzNh5PZz93UChVHyxOu+m3f/G+3i+Eeg9703zSF4DRwukQUoEGYzfaLdeSjFaLfiw0hKDgesxbsAgw6tC/YjmGDhvBhBdeSmffu1d3khIT2fj51tS0xo0aEBhYm4VLlqGUolJ5P0aOGceYseMBuHLlCg+V9eHd91bRrXuPTGl2RY3JN1Nspoc91pC6QcHMmbcwVV91/4cYPHQ4Yyekvw/79e5BUlIiazd+kZrWvHEotQIDmbdwqc0yEhISKOdTgs+3fUPTZs1t2lxNvmUzHaBtWCNq1w3mtTnzUzXWrV6JgYOfZ8TYF9LZD+7Xi6SkJFav3Zya1q75o9SsFcib8xbbLKNfr8789+9/rP9ie7p9/yYkUKWcF4CnUirBrlA7mJ8FNV78nHwFPBzNzq3riZx4o5PT5WcnuoXjgiQnJ3P86BGaWHz53NzcaNKsOVGRB23miYo4aGUPENaiFVERt+1TUlIYOrAvI0aPo1r1Gnel8YH8btSp7MWu43+mpikFu374k/pVvW3mOfhLDHUqe6V2u1XwKUrroHJ8ffiClZ1/mWL89l4vflranQ9GN6Ocl+NfwuTkZI4eOUxY8xapaW5uboSFtSDyYLjNPBEHw2kW1sIqrWWr1kSY7M+dPUt0dDRhFjaenp7Uqx+SanM/aUxOTubY0cM0DbO+D5uGNScq0vaxoiIO0rSZtb7mLa3vw7RlrHpvBZ6entSqFeiQPnP+H44doXHTMCuNjzUN41CU7TIPR0VY2QM0bd7Srn1cbAw7tn9Frz79HNansSZPRamJ0fZcDnQBSgB1lFLHclZVei5diufWrVuU9rZ+cJf29ubUqV9s5omNicbb28cqzdvbh9iY6NTP899+k3z58zPk+RF3rdGraEHy53Mj9spVax3/XKVq2eI286z57ldKFSvIzlc6ICI8kN+Nd7/+ibc23L4EUadjGbxwL6f+vIJvicJM7l6XHa90IGjUBv67lvn3jsTHG3WYrk58fDh50nYdxkRH4+2Tvg5jTHUYHR2deoy0x4yxqOf7ReMlO/pKe/tw6uRJ2/piovFOd9+mL/vrbVsZ0Mdoafj6lmHT1u2U8vJySB/A36nflTQaS3tz5pRtjbEx0em/W6V9iI2JsWm/5pOPKVKkKO063LvuNNBh0fcjbYB+QFPgNyA+J8VkJ8eOHmb5koXsPhCVY32+j9Uow4TOtRn17vdEnYqlchlP5gxsyF9dk3h93VEAvjnyR6r9j7//TdSpWE6+25POjSrx4U7bDxBN7uOxJs34LuIIf8fHs+qDlfTr3YNd+8LTOQJX4LPVq3iqW08KFrQ9NplVCE6GReei1TvzWpdaZeAvpdQBpVS0UupmVhcgIu53e4xSpbzIly9fuoHZuNhYfHx8bebx9vElNtb6F1psbAzeJvvw7/cTFxdLQNWKlC5WgNLFCnDh/O9MnTiBwGqVHdYY/+81bt5KwduzkLWO4oWI/ifJZp7pvYL5dO9pVu04yYnzl9kScY5p/4tiQufadn+lXUlK5szFK1Qu49hbUb28jDpMVycxMfj62q5DH1/fdL9yY2NjUuvcnC+dTUyM3euSmzWWsqMvLjYGH18fm3l8fHzTBTzExaYv28PDg8qV/akX0oDFy1aSP39+PvrQ/nihPUqmflfSaIyLTdfKM+Pt45v+uxUXY9P+4IH9nDl9iqf79HdYm6PoKLX7CBFZBSwEyouIEpFzIuImIhNF5KyIXBWR4yLSxSJPPhF5z2L/SREZlfa4IrJZRCaLyEXgrn+Gu7u7E1inLvv27EpNS0lJYe+eXdSr38BmnnohDazsAfbs2kG9EMO+e8/efBdxlL3hh1O3MmX8GDF6HOs/tx8+bY8bN1M4+ms8zQLKpqaJQLNafkSetB0dVKhAPlLSjE2n3FKmvLa/NR4F81PRtyjRl207MXu4u7tTp24Qu3ftvF1WSgq7d++kfoOGNvOENGjInt07rdJ27viWEJN9hYoV8fX1ZbeFTUJCAlGREak295NGd3d3atcJYu/uNPfh7l3Uq2/7WPVCGrB3j7W+3Ttv34f2SElJIfn6dYf0mTUG1K7Ld3t3Wx1r/97dBNezXWZQvRC+22v9Xdm3e6dN+08+/oCA2nWp4cT4kqPkhSi1vNSlNgr4FRgM1ANuAROB3sBzwGmgMbBaROKUUnsxHPIfQFfgEhAKvCsifyml1locuzmQALS0V7iIFAAKWCQVzUjs8yPGMGxwf2rXCaJucD2WLV5AUlIivZ7pB8DQZ/tRxs+PabNeBWDI8yPo0DqMRfPfoVWbdmxcv4ZjRw4zd+EyAEqWKkXJUqWsysj/wAN4+/hS5eGqGUmxy4It/8eKkU04/Gsch07HMfzxmhQu+AAf7TwFwMqRTbn4dyLTVhtzIrZFnWdkx1ocPxtP5Kk4KpcpxrReQWyL+p2UFMPxvNY3hC8P/c752P/wK1mYKT2CuJWiWPvdrw7rGzl6LIMG9CUoKJjgevVZtGAeSYmJ9Olr/Fod2K8PfmXLMvuV1wAYNnwUrZo3Yd7ct2nbtj3r1n7GkcOHWLz0XcB4IAwbOZo3Xn0Zf/8qVKhQkZkzplLGz4+OnZ5wqg5dXeOwkaMZOqg/dYKCCAquz5JF80lMSqS3aQB9yMC+lPEry4zZxn04dNhI2rVqxsJ579C6bTs2rFvD0SOHmL/YuA8TExOZ88artGvfAR/fMly6FM/K5Uv46+KfPPFUF3syMmTIsFGMGjqQwDp1qRNUjxVLFpKUmEiP3n0BGD6kP2XK+DF5xisADBo6gifbNWfpwrm0aN2WzRvWcvzoYd6av8TquP8mJPDF5g3MePlNp3Rp0pNnHI5S6oqI/AvcUkpFmxzAJKCFUsoccvObiDwKDAH2KqVuANMtDnNWRBoC3QBLh5MIPKuUSs5AwsQ0x8qQp7p041J8HK+9PIPYmGhqBgSybvOXqc3+P/44j5vb7QZqSINQ3v1gNa/OmsbLM6ZQqXIVVn+2geo1ama2SIdZ//1veBUryLQeQfiUKMwPZy/RadZXqYEE5Up7kGIRdv/6uqMoZXSt+ZX0ID7hGl8e+p0Zqw+l2pQt5cFHY8MoWbQg8VeucuDnGJq89DnxCY7PIenarTvxcXHMmjmNmOhoAgJr8/nWr/Ex1eGFC9Z12DA0lFUff8LM6VOYPmUS/lWqsHbDZmrUvF2H48a/QFJiIsOHDuaff/4htNGjbNn6tdP9+66usXPX7lyKj+fVWTOIiYmmVkBtNn6+7fZ9eOGC9X3YMJSVq1bz8sxpzJo+mcr+Vfhk7cbU+zBfvnycOvkLn67+iEuX4ilZshR1g4P5asdepyMnn+jcjUuX4nnz1VnExURTo1Ygn27cmhpI8Ocf1hrrhTRkycqPeOPl6bw2ayoVK/vzwSfrqVbd+ruyecNaUIonu3R3Spej5IWggTw1D0dERgOjlVIVRKQG8COGs7DEHTiqlAox5RkGDADKA4VM+48ppeqb9q8Cyiql7LZuTHa2Wjh/2JuH4ypkNA/HFchoHo4m89ibh+MqZDQPJ6fJqnk4daZsJV9BJ+bhXEvk6MuPO11+dpJnWjg2KGL6vz3wZ5p91wFEpAcwBxgHhAP/AhOAkDT2aZ1WOpRS183HNR3bKdEajeb+JC+0cPKyw/kJwwGUN43X2KIRcEApldq5KyKOh3RpNBqNJu86HKXUvyIyB5grIm7AfsATw8kkKKU+xAgk6CMirYGzwDMYAQdnc0i2RqO5T8kLryfIsw7HxFQgDmNAvxLwD3AEeNW0fzlQB1gDKOBTYAnQNtuVajSa+xtn59TkHn+TtxyOUmoeMM/iswLmmzZb9teB/qbNkokWNv2yXKhGo8lz5IUWTp6Z+KnRaDSanCVPtXA0Go3GVdFRahqNRqPJFvJCl5p2OBqNRuMC6BaORqPRaLKFvNDC0UEDGo1Go8kWdAtHo9FoXIC80MLRDkej0WhcAD2Go9FoNJpsIS+0cPQYjkaj0WiyBd3C0Wg0GhcgL3Sp6RaORqPRuADmLjVnNifLGyYi50TkmohEiEj9O9gXF5HFIvKXiFwXkVMi0s6RMnULJ4cp5J6PQu75clqGXVz9jZolQsfltIQ7ErvvrZyWcEceyOfaP5MfKOTCj6obWaNNcLKF40xZIt2Bd4DngAhgNLBdRKoqpWJt2LsD3wKxQBeMl1Y+hLHCfqZx4auo0Wg0mnvEWGCFUuoDABF5DuPtxwOA123YDwBKAqFKqRumtHOOFqq71DQajcYFcBNxejNRVESKWWwFbJVjaq0EATvMaUqpFNPnhnbkdQTCgcUiEiMiP4rIJBFxqHtGOxyNRqNxAcxBA85sJv4ArlhsE+0U5QXkA2LSpMcAvnbyVMLoSssHtANmA+OAKY6co+5S02g0GhcgC+bhPAj8a7HrehbIMuOGMX4zWCl1CzgsImWBCcDMzB5EOxyNRqO5P/hXKZWQCbt44BbgkybdB4i2k+cv4IbJ2Zj5GfAVEXelVHJmBOouNY1Go3EB3MT5zRFMzuEw0NycJiJups/hdrJ9D/ib7Mw8DPyVWWcD2uFoNBqNayDOzcVxKi7aCIkeJCJ9RaQasBTwAMxRax+JyGsW9ksxotTmi8jDItIemAQsdqRQ3aWm0Wg0LkB2rjSglFojIqWBWRiBAseANkopcyBBeSDFwv6CiLQG5gI/YMzDmQ+84Ui5mXI4ItIqswdUSn3jiACNRqPRgJj+OZPPGZRSi4BFdvY1tZEWDjRwqjATmW3hfJ1JO4URNqfRaDQajRWZdTiF7qkKjUajyeM4EwBgzpdbyFTQgFLquq0NI0wubZomi1i2ZDFV/StQvEhBHgsNISoyMkP7DevXEVjzEYoXKUhw7Vp8/dU2q/1KKWbNmEbFcmUoUbQQ7Vq34Mzp0/e1xiFdGvHL5slc/u519r0/kuDq5eza5s/nxsSBLTmxcSKXv3udiP+No2WDqlY2kwe14mrk21bbsbUvOq0P4N1lS6hZtRKlixem2WMNORSVcR1u2rCOoMDqlC5emAbBgWz/2roOX315JkGB1fEtVZTyZUrRsV0roiIjnNa3bOliHqlSkRJFC9G4UQOi7qBv4/p11K5ZjRJFC1GvTkC6a7x500Y6tGvNg75eFHZ34/ixY05ry00a70R2L96ZEzgcpSYibiIyQUR+Ba6JSCVT+nQR6ZPlCvMo69au4cUJY5k8ZTrhkUcICAikY/vWxMamW1cPgPADB+jbuyd9+w/kYNRROnR6gm6dn+DEjz+m2rw9502WLFrAgsXL2Pd9BB4eHnRo35pr167dlxq7tKjNG6M78srKb2jYZy4/nL7IlgWDKV2iiE37GUPb8uyTDRk7ZxN1ur/Jyo0HWPNmfwIfLmtld+LXv6jQdkbq1nyQzW7wTLFh3RomvTiOlyZP5bvwQ9QKCOCpjm2Js1OHEeEHGND3afr0HcD+g4dp36ETvbo9xU8nbtehv38V5sxdQPih42zfuY/yDz3Ekx3aEB8X57C+9WvX8NKEcUyaMo0DEYepFRBAp/Zt7F7jg+EH6PtML/r2H0B45BEe79iJ7l2etLrGSYmJNAxtxOxXbS3Z5Ti5QWNmyIKVBlweUUo5lkFkIjAEY2mDhUBNpdRvItITGKGUCs16mfcfIlIMuBJz6QrFihVLt/+x0BCCgusxb4HxMEtJScG/YjmGDhvBhBdeSmffu1d3khIT2fj51tS0xo0aEBhYm4VLlqGUolJ5P0aOGceYseMBuHLlCg+V9eHd91bRrXsPh8/BFTRmtFr0vvdHcvinC4yZswkwfkGe+WIqS9fuZ85Hu9LZ//blNN74YCfL13+fmvbp6325ev0GA6Z/AhgtnA5NatKg9zuZqSIg49Wimz3WkLpBwbw9byFg1GE1/4cYMnQ4Yyekbzn1692DxKRE1m38IjUtrHEoAYGBzFu41GYZCQkJPOhTgi3bvqFps+Y2bfLbWS26caMGBAUHM3f+7WtcpVJ5hj4/nPE2rvEzvQx9Gzff1tfk0YYEBAaycPEyK9vfz52j2sOVCI88QmDt2jbLzww5rTEhIQFfr+IAnpmceGmF+VnQbsFuHihk+8dQRty4+h/bRjZzuvzsxJl5OP0xljd4D2O2qpljwCNZoiqPk5yczNEjhwlr3iI1zc3NjbCwFkQetD0vK+JgOM3CWliltWzVmgiT/bmzZ4mOjibMwsbT05N69UNSbe4njQ/kz0edRx5kV9Tt7jilFLuiTlG/1kM287i75+da8g2rtKvXbxAaWNEqzb+cF799OY2fNk3ig1lPU86nuEPazCQnJ3Ps6GGahd12Am5ubjQNa05kpO3zjYw4SNNm1nXYvGUrIiMO2i1j1Xsr8PT0pFatQIf1HT1y2Oqama9xxEHb5UVEhBMWZu3UWrRsRaQd+7slN2jMLFmweKfL44zDKQecsrPP5uqkGseIj4/n1q1beHtbrzzh7eNDdLTtlSdioqPx9klj7+1DTIxhb86Xzsbnts39pNGruAf58+cj9u9/rdJj//4P31JFbebZcfAkI3s1oXI5L0SEsPoP06lZLXy9brdAo348z+BZn9Fx1ApGvrGBCn4l2fHuMIoUdvzWv2Sqw9Jp69Dbh5jotOsqGsTEROPt7Z3ePk39fLVtK2W8ilG6eGEWL5zH5q3bKeXl5ZA+8zX2SXfNvO1ej5joaJv3hDP32P2iMbPkhS41ZxzOSWwvYf0kxoSgXIuIrBKRzTmtQ5MzjH97M79eiOf42hdJ+P4N5k54ko++iCIl5Xa38zfhv7Bx5w/8eOYvdhw8yROjV+BZtBCdWzjWerjXNG7SjP0RR/h2935atGpNv9497I4LaVyDvBA04MxKAy8Dy0XEG8NhtRORqsAgDKeTmxmFswtFZCFeXl7ky5eP2FjrX7mxMTH4+tpePdzH15fYmDT2sTH4+Bj25nyxMTGUKVPG6pgBgY73n7u6xvh/Erl58xbeJa1bM94lixB96V+7ebpN+IAC7vkp5VmYi3EJvDy8PWcvXrJbzpX/rnHmfByVH3Ss9QBQylSHcWnrMDYGH9+06yoa+Pj4phsMt6xDMx4eHlSu7E/lyv7UD2lA7ZpV+ejD9xk3If2Yhj3M1zgm3TWLTVdeqj5fX5v3hD37uyU3aNTcxuEWjlJqPdAd490IN4F5GC2erkqpr7JWXvailLqilHLolan3And3d+rUDWL3rp2paSkpKezevZP6DWy/HymkQUP27N5plbZzx7eEmOwrVKyIr68vuy1sEhISiIqMSLW5nzTeuHmLo7/8QbN6VVLTRIRmwVWI/L/fM8x7PfkmF+MSyJ/PjSeaBbB17492bT0KuVOxrBfR8Y6P1bq7u1O7ThB7dt8OYEhJSWHv7l3Ur2/7fOuHNGDvHus63L1zB/VDMp4AnpKSwvXrjs1aMF9jy2tmvsYhDWyXFxLSkN27rAMydu3cQX079ndLbtCYWfJCl5pTa6kppXZgeluciIhyNNTNRRGRVUBxpdQTprflvQX0AIoBh4AxSqkoMdqwp4FlSqk5FvlrA0eBKkqpM2mOXQDrMS7bAwkmRo4ey6ABfQkKCia4Xn0WLZhHUmIiffr2B2Bgvz74lS3L7FeM9fWGDR9Fq+ZNmDf3bdq2bc+6tZ9x5PAhFi9911w+w0aO5o1XX8bfvwoVKlRk5oyplPHzo2OnJ5yqL1fXuOCTfayY3oPDP1/g0InzDO/RmMKF3PloqzFHY+WMnlyMvcK0JcYcjHo1yuNX2pPjp/6krLcnkwe1xs1NeOfj3anHfG1kB7787gTnoy/j5+XJlMGtuZWSwtpvjjpVh8NHjua5Qf2pExREcHB9liyaT1JSIr379ANg8MC++PmVZcbsVwEYOmwkbVs1Y+G8d2jdth3r163h6JFDLDBFVyUmJjLnjVdp274Dvr5luHQpnhXLl/DXxT958qkuDusbOWoMgwb2o25d0zVeaFzjZ0zX+Nn+ffHz82OW+RqPGEmr5k2ZP/dt2lhc40VLlqce8++//+bC+fP89ddFAE6fOgkYLQ97rePcrjEzOBsAkJuCBpxevFNEagLVTH//pJQ6kWWqXIM3gc5AX+B34AVgu4j4K6X+FpH3MSL25ljk6Q/sS+tsTEwEpme28K7duhMfF8esmdOIiY4mILA2n2/9OnVw9MKF87i53W6gNgwNZdXHnzBz+hSmT5mEf5UqrN2wmRo1a6bajBv/AkmJiQwfOph//vmH0EaPsmXr1xQsWDCzsnKVxvU7juFVwoNpg1vjU6oYP5z6k06jVhD7938AlPMpbjU+U8A9P9Ofa0PFsqX472oy2w/8zMDpn3Dlv9tzgMp6e/LRy70p6elB/OX/OHD8LE0GLCD+n0SH9QF07tqd+Ph4Xp01g5iYaGoF1GbD59tSAyf+uHDBqg5DGoby3qrVzJ45jZnTJ1PZvwqfrN1I9RpGHebLl49TJ3/hk9UfcelSPCVLlqJucDBf79hLteo1HNbXpVt34uLjmD1reuo13rz1K7vXuEHDUFZ99D9mTp/K9KmT8fevwpr1m6yu8ZdbtzDk2QGpn/v07gnApCnTmDJtxn2pMTMIzvXn5x5349w8HF/gY4x3J1w1JRcEdgPPKKX+ylKF2Yi5hQM8DVwG+imlPjHtewA4B8xTSr0lIn7AeSBUKRVp2n8RGK+U+tDGsW21cP6wNw9HkzkymofjKmQ0D8dVsDcPR3NnsmoezlNL9zk9D2fj0MZOl5+dOBOlthIoAdRRSnkopTyAuoAnsCIrxeUglYEHMF46BIBS6gYQialVp5S6CHwJmH8GdcBwKOtsHdC09E+CecP6VbAajSaPkxei1JxxOM2BIUqp4+YE09/PA2FZJSyXsBLoISKFMLrT1iilknJYk0ajyYVk1xs/cxJnHM5FO+kK++/Dzm38CiQDjcwJpi6zesBPFnbbgERgKNAGeD8bNWo0mvsI3cKxzUvAQlPQAJAaQDAPuLtlc10EpVQixitV3xKRNiJSHaO7sDDwnoXdLWAV8Bpw2vSCIo1Go3GK+zkkGjL/xs+/MFowZkoAx0XEHDRQCKNFMB87Yxi5kJcwHPLHGAP8h4DWSqnLaezew3i39wfZK0+j0WhyF5kNi55xL0W4EAWA/wCUUteAkaYtI8oCN4CP7q00jUZzP+Ns91hu6lLLlMNRSi2/s1XuRUTyAw9jrJiQqXM1hTmXxnDG65RStldb1Gg0mkyg3/h5B0wvY3O33LJKWDZTE6PL7ASw7A62ZnpiTAgtjjEpVKPRaJwmLwQNOLzSgCkEeDbQDfAj/UTXfFmgK1tRSh3DCAhwJM8qjIABjUaj0WQCZ1o4rwEdMZZqSQaGmdJiuD0JUqPRaDQOIHex5RacWUvtSWCAUmqniCwDdiilzojIrxhrj6Vb1kWj0Wg0GZMXFu90poXjhbFSMkACRog0wB6gWRZo0mg0mjxHXng9gTMO5yxQ3vT3SeAp09+tMRyQRqPRaDTpcKZL7WOMJV72Y7wvZrOIDAM8MMZ1NBqNRuMgeh6ODZRSb1j8/ZVpWZt6wBmlVGRWitNoNJq8grPdY7nI3zj/AjYzSqnT3B7T0Wg0Go0T5IWggcyupTY4swdUSr3rvByNRqPJm+gWzm1mZtJOAdrhaDQajSYdmV1Lrcy9FqJxTRx9BXl2E7FhRk5LuCMVn1ub0xLuyPl3u+e0hAzJTb/inUUHDWg0Go0mW3DDuXkqd7UgZjajHY5Go9G4AHmhhZObnKNGo9FocjG6haPRaDQugDj5Ppxc1MDRDkej0WhcAf0CNjuISH0RWSkiu0XEz5TWQ0QaZK08jUajyRvkhRewOexwRKQjsBcogPFK5oKmXd7AlKyTptFoNJr7CWdaONOB4UqpZ4AbFun7gaAsUaXRaDR5DHOXmjNbbsGZMZxHgJ020v/h9rtxNBqNRuMAemkb28QCFYFzadIbYrwrR6PRaDQOkhcW73SmS+0DYJ6IBGKsnVZKRDoDc9DrqGk0Go3GDs44nJeBLUA4UAQ4CHwCrFZKzc1CbXmeZUsWU9W/AsWLFOSx0BCiIjN+3dCG9esIrPkIxYsUJLh2Lb7+apvVfqUUs2ZMo2K5MpQoWoh2rVtw5vTdvVli2dLFPFKlIiWKFqJxowZERWWsceP6ddSuWY0SRQtRr06AfY3l/ShZrDDt27S8K42fffgubUNrUq9KaZ7u2Iz/O3bIru2Zkz8zdkhv2obWJLB8MVavXJzO5tatWyyaM5u2jWpRv4o37R8NYPn8N+5qzbkBzf05MqcDf6zoyvapLalTsWSG9kNaPczB19px4d0uHH+7Iy/3rEOBB6y/yo4eMyOWL11MtYcrUrJYIZo82oBDd7rGG9ZRp1Y1ShYrRL266a/x55s30qFda8qV8cKjgBvHjx9zWpsZV78PM4PbXWy5BYe1KqVSlFJTgdJAMNAM8FVKTchqcXmZdWvX8OKEsUyeMp3wyCMEBATSsX1rYmNjbdqHHzhA39496dt/IAejjtKh0xN06/wEJ378MdXm7Tlvsq88V70AACAASURBVGTRAhYsXsa+7yPw8PCgQ/vWXLt2zSmN69eu4aUJ45g0ZRoHIg5TKyCATu3b2NV4MPwAfZ/pRd/+AwiPPMLjHTvRvcuTVhrfmfMmSxcvZMGipezdf5DChT3o+HgbpzR+vWUDc2ZPYsjol/jsy++oWq0WQ3s/xaX4OJv2164l8WD5Cox8aQZepX1s2nywdC7rPn6PibPeYtOuKEZPnMWqZfP55INlDusDeKJ+OWb3qMNbm38kbPp2Tlz4h3Xjm+JVtIBN+84NHmJq10De+vwEoZO+YtT7kTxRvzxTOgc4fcyMWL9uDS+9MI6Jk6fxfcRhatUKoNPjGV/jfs/0ok+/ARyIOEKHjp3o0fVJTpy4fY0TExMJbdSI2a+87rAemxpd/D7MLOYxHGe23IK4+mrA9ysiUgy4EnPpCsWKFUu3/7HQEIKC6zFvwSIAUlJS8K9YjqHDRjDhhZfS2ffu1Z2kxEQ2fr41Na1xowYEBtZm4ZJlKKWoVN6PkWPGMWbseACuXLnCQ2V9ePe9VXTr3sOmzozuj8aNGhAUHMzc+bc1VqlUnqHPD2e8DY3P9OpBYlIiGzd/kZrW5NGGBAQGsnCxSeNDZRk1eiyjLTRWeNCXd1d+QFcbGk9H/2dX39Mdm1EjsC6TZr+dqq9VSDV69hvCwGFj7eYDaBtak6cHDKX3s8Os0of360qp0t7MfOt262fskN4UKFiQ1+avtHmssKlf2i1n+9SWHD17iZdWHwGMh8cP73RkxY7TLPjy53T2r/euy8N+njz15u7UtFk9alO3Uikef3WnU8cE+6tFN3m0AUFBwbxjcY0frlye554fzvgJ6a9xn6d7kJiYyAaLa9z0sYYEBASyYLG1U/793DmqV63EgcgjBAbWtltH5nOwR07fhwkJCfh6FQfwVEolZHgiNs/NeBZMWH+EAh5FHM3O9cT/eKtLXafLz06cmYezLaPtXojMayQnJ3P0yGHCmrdITXNzcyMsrAWRB8Nt5ok4GE6zsBZWaS1btSbCZH/u7Fmio6MJs7Dx9PSkXv2QVBtnNFqWadYYcfCgbY0R4YSFNbdKa9GyFZEm+3NnzxITHW11zFSNEY5pvJGczM//d4wGjzaz0tfg0ab8cMT5N6HXDg4h8vu9nPvN6F45+dP/cTQqnEebtnT4WA/kcyOwQgn2/hSTmqYU7D0R8//snXd8FNXXh58TqlQRSAIC0hGkhBp670izgA2QYqEICIpY6PYCCFhQeVEsP+lFFFGaDQhIVQTBAqKYhA4GECTn/ePOhk2yCewm2d2Q+/CZD9mZO3e+OzM7Z+65555L3XKFPe6z+Zej1ChdKMFFdkPRvLSuXoxVO//2uc6USOkat2jZOuGaJSUqagMtPFzjqCjP5dNKsN+H3pAVWji+RKkdSPI5BxABlAf+l2ZFGYSIrAO2q+rwQGu5HEeOHOHixYuEhiZ264SGhfHzz3s87hMTHU1oWJLyoWHExEQDEB0dnVBH0jpdZXzRGJbsmKGpa/TwnVzHd/3v8XtEx+ANx48d5eLFixQuUjTR+sJFQvn9171e1eVOv0Ej+Of0abq1qEO2bNm4ePEiDz06lk7dvZ9PpnD+nGTPFsLhk4ndNIdPnaNCseStXoCFGw9wXb6cfPpkKwQhR/YQZq/Zx9TlP/lcZ0ocdd2HHq7xXh/vw/Qm2O9DS2K8NjiqOtDTehF5FshEttZi8Z6Vyxfx2ZJ5PDd9FuUrVmbPrp28NGE0RcPC6XL73Rl+/EY3hjK8cxVGzdnClt+OUiY0H8/eXYuRXc7xyrJdGX58S8Zhc6l5x2zgvnSsL8tSpEgRsmXLRmxs4rep2JgYwsPDPe4TFh5ObEyS8rExhIWZ8q79kpWJuVTGF40xyY4Zm2J9YeHhHr+Tq7zrf4/fI9xzJ35KFLquMNmyZUsWIHD0SGyKAQFXwpRnxtBv0MN06HIbFW68ic633sk9AwYz6/XJXtd19PR5/rsYT9GCuROtL1ogN7Enz3rcZ3T3asxfv58Pvv6N3X+e5LOtf/HMgp0M61QZEd/qTInCrvvQ22ucyn2Y3gT7fegNJlu0eL1kJpdaehqcWiROdRO0iEghEZkjIsdF5IyIrBCRCs62AiJyVkQ6JNmnu4icFpE8zueSIjJPRE6IyDERWSoipdNDX86cOalZqzZr11xK6BAfH8/ataupV7+Bx30i6zdg3drECSBWr/qSSKd86TJlCA8PZ61bmVOnTrF5U1RCGV80uh/TpTGyvuccrpGRDVi7Zk2idWtWr6KeU750mTKEhYcnqjNBY6R3GnPkzEnlahFEfbcukb6o776ieq16XtXlzrmzZwgJSfyzyRaSjfj4eK/runAxnh37j9O0yqWHmAg0rRLG5l+PetwnT65sJD3UxXgT2CGIT3WmRErXeN3a1QnXLCmRkQ1Ytzb5NY6MzJi8vsF+H3qD7cPxgIh8lHQVUAxoBLyYHqL8wLtABaALcAp4AfhMRKqo6ikRWQ7cBaxw2+duYImqnhGRHMBKzFikJsB/mMSln4tIdVU9n/SAIpILk/DURf7UBA4dPoL7+vWhdu061KlbjxnTpnImLo7effoC0P/e3hS//nomPfMcAIOHDKNtq2ZMnfIKHTp0Yv68j9m65Xtee+Mt1/EZPHQ4Lzz7NOXLV6B06TJMGD+GYsWL06VrNy9Pn6Nx2MPc1/9eatVyNE43Gns5Ggf07UPx4sWZ6NL40FDatmrOq1Neob2bxhmvz0zQOOShYbzw3DOUczROHD+WYsWL09kHjb0GDGHMyAe5qVpNqkbU4YNZr3P2zBm69bgHgCeH309oeHGGjR4PmECDX/ftSfg7NuZv9uzaSZ68eSlVuhwAzVp34O3pLxNevATlHJfa++/MoGuPXj6dwzdW7mHGffXZ/vsxtv52jAfbViRPruz875vfAHjtvkj+Pn6WpxfsBGDl9kMMbFeJH/44zpZfj1ImLB+jb6nGF9sPEe9EFF6uTm94aNjD3N//XmrWrkOdOvV4zXWNezvXuJ9zjZ8213jQkKG0a33pGi+Yb67xdOcaAxw7doyDB//g70OHANi392fAtCxSasGnRrDfh5ZL+BI0kNSexgPbgcmquiztkjIWpyXTBWikquuddXcDB4FuwHzgQ+B9EcnjGJgCQCegu1NNT0zrcIA6ccMi0heTT6458IWHQz+OSXx6RdzeoydHDh9m4oSxxERHU71GBEuXf57QOXrw4B+J3rQbNGzIu+9/xIRxTzHuqScoX6EC8xYu4aaqVRPKjHxkFGfi4hgy8H5OnDhBw0aNWbb8c3Lnzp3s+FfCbT16cvjIYSZNHJegccnyFSlqrN+gIe/O+ZAJ48YwbsyTlC9fgbkLFifSOOKRUcTFxTFk0AOcdDQu/WSFTxrbd7mV48eO8PrkZzlyOIZKVarx+vsLKVw0FIDoQ38m0hcb8zc9OzRO+PzezGm8N3Madeo3ZtY8E4A5euJLvPby0zz71EiOHTlM0bBwbru7Lw8MSx5+eyUs2XSQwvlzM7p7NUIL5ubHP07Q45V1HD71LwAlCucl3i0y/ZVlu1BVHr+lGsUKXcPR0/+ycvshnlm484rr9Ibbbjf34dPu1/iTS9f4Tw/XePacD5k4bgzjxz5JufIV+Hj+Ym666dI1/nT5Mh68r1/C5z733AnAE0+N5ckx473XGOT34ZWSFfpwvBqHIyLZMBmhf1bVkxmmKgNwRakBa4CFQG5Vvei2fRuwWFUnikhOIBoYpKofO8bkeeB6Vf1PRF4CHgaSjgLLAwxW1Tc8HN9TC+fPlMbhBAvBPk4rtXE4wUJq43CChZTG4QQLwew2Sq9xOGOWbiN33lQdHx45F3eaSV1r+nx8f+JVC0dVL4rIN0BlIFMZHG9Q1fMisgDjVvvY+X+uqv7nFMkHbMG42ZLicRi7qv4LJLxiZqZJkywWS8aTFVo4vgQN/ASUTG8hfmQ3xtBGulaISGGgEua7ufgQaC8iNwEtnc8utmL6gGJV9Zcky1VriC0WiyUt+GJwRgEvi0hrJ9orp/uS3gLTG1XdBywF3haRxk7W6w+Av5z1Lr7GuNU+BH5X1Si3bR8CR4ClItJERMqISHMRmSYiJfzzTSwWy9VEVpiAzReDsxLTj7MS89A9m2TJDPTFuMSWYyLNBOioqglh3U4wwP+AGiRu3aCqZ4CmwB/AIkyraRZmuu2g9qFaLJbgRER8XjILvkSpdbh8keBDVZu7/X0c6H0F+zwGPJbCtmigT3rps1gsWRt/9+GIyGDgUSAc2AE8pKqXTTQoIndgXsaXqqpXceJXbHBEZCzwsqqu9OYAFovFYgkuRKQnMBl4EIgChgMrRaSSqnqe18HsVxoz2eY3vhzXG5faOEx0lsVisVjSmXTINJDfyZTiWlKbAGkE8LaqzlbVnzCG5wzQL6UdnGExH2JsgfejiPHO4GQeR6HFYrFkMnzJo+ZaHP7EDFdxLY97Oo4T3FUbWOVap6rxzufUcveMxUTmzvL1O3rbhxPcowAtFoslk5IOfTglgNNum1JKLVEEyAYknWshBrjR0w4i0hjoj5mKxme8NTh7RSRVo6Oqvk+ebrFYLFkVXxNxXtrndEZkGhCR/MD7wH2qeiQtdXlrcMZxFWcYsFgslizAEeAikHSuhTDM2MOklANKA5+4hWCHAIjIf0AlVf31Sg7srcH5OLUIBovFYrH4RghCiA9d5d7u46Tu2gK0ApYAiEiI83mGh132ANWSrHsakw9yGCbx8RXhjcGx/TcWi8WSQfg6t42P4z4nA++JyPfAJkxYdF7MRJqIyBzgL1V9XFXPAT8mPqacAFDVROsvhzcGx0apWSwWSwbhz4GfqjpXRIoCEzEDP7cD7VXVFUhQCjP1TLpyxQZHVdNzdlCLxWKxBBBVnYFnF1qizCwpbL/Xl2P6ktrGYrFYLOlMkjE1Xu2XWbAGx2KxWIIAP/fhBARrcCwWiyUICMHHFk4m6l63BifAxMcr8fE2ANBXKoQHf3q/P9+5I9ASLkuhukMCLSFVjm/22NUQFGSm6QECjTU4FovFEgRYl5rFYrFY/EIIvs2ImZnCh63BsVgsliDA19k7M5NLzxoci8ViCQIE30bXZx5zk7laYxaLxWLJxNgWjsVisQQBduCnxWKxWPxG5jEdvmENjsVisQQBWSEs2vbhWCwWi8Uv2BaOxWKxBAFZISzatnCCmJlvvEblimW4rsA1NGtcn+83b0q1/KKF86lZrTLXFbiGurWq8/mKzxJtX7pkEZ07tqNksSLkzRXCjh3br3qNb77xGjdWKEOh/NfQtFF9Nl9O34L5RFStTKH811C3ZnJ9SxYbfSXCi5AnZwg7tqf9HL75+mtUKl+aa/PlpknDSDZvSl3jwgXzqVH1Rq7Nl5s6EdWSaVRVJo4fS5mSxSiU/xo6tmvNL/v2+azvgR5N2fPpBI5vnMLXcx6hzk03pFg2e/YQHr+/PbuWjeP4xilEzR1Nm4aVE5XZ8+kEzm6bkWyZMrqHzxqD/RxeCSFpWDILmUlrlmLB/LmMHjWSx58cy3dRW6hWrTpdb25PbKznGb43bljPvb3uove9/VgftZXOXbpyx+3d2bXr0oR8cXFxNGzUiEnPPJ8lNC6YN5fRj47kiafGsj5qC9WqV6drp9T19el1F3369mPDpq3c3KUrPW/rzq4fL+k7ExdHg4aNmPRs+pzD+fPm8tijI3jyqXFs2LSV6tVr0KVTuxQ1bli/nj733Emfvv3ZuHkbnbt2o8et3RJpfOXlF3l9xjSmvfYmX38XRd68eencqR3nzp3zWt9tbWvxwsjuPDNzBQ3ueoGde/9i2euDKVrIcw678YM6M+DWxox4cT41b32adxZ8y9xX7qNGpRIJZRrf8xKlWz+esHR8cDoAi77c5rU+CP5zeKW4Wji+LJkFUbWJIwOBiBQATv59+AQFChRItr1Z4/rUrl2Hya+apIXx8fFULFeKBwcN4ZFHRycr3/vuO4iLi2Phkk8S1jVv0oDq1Wsw7bU3E5U9sH8/VSqVZf2mrdSoEeHzdwgGjan91po2qk/tOnWY4qavQtlSDBw0hEdGJdfX6647iDsTxyI3fc0aN6B6jRpM96CvcsWybNi0lRoRqZ/D1B4ITRpGUrtOXaZOu6SxfJmSDBz8EI960HjPXT05ExfHoqXLE33PGjUimP76m6gqZUsVZ+jDI3l4xCMAnDx5khuuD+OtWe/So6fnRKIpJe/8es4jbNl1gIdfmJ/wXX75fBJvfPwVL8/+Mln53754hhfeWcnMeV8nrPvfywM4e+48/Z6a4/EYLz1yKx2aVKVq1wket0PqyTsDfQ5PnTpFWOGCAAVV9VSKQlPA9SyY/c0e8uTL7+3unPnnNH2b3Ojz8f2JbeEEIefPn2fb1i20aNk6YV1ISAgtWrZm08aNHveJitpAi5atEq1r3aYtUVGey1/tGlPS17Jla6JS0dfSg76Uvk96aWzZKrnGTRs3eNa4cUOi7wTQpm07opzy+3//nejoaFq6lSlYsCB160UmlLlScmTPRs3KJVkT9XPCOlVlTdTP1KtexuM+OXNk59z5C4nWnT13noY1y6V4jDs61uW9pd5pcxHs59AbJA1LZsEanCDk6JEjXLx4kdCwsETrQ0NDiYmJ9rhPTHS0h/JhKZa/2jUecfSFeasvNEn5sIw7hy6Nno4ZHe3bOXTtl6yMD9+jSKF8ZM+ejdhjpxOtjz16ivDCyVvlAKs27GboPS0pV6ooIkLLyBvp2jKC8CKey3dpUZ1r81/DB59EeaXNRbCfQ2/ICi61q8rgiIiKSLdA67BYsiqPvLSAX/+IZceiMZzaNJUpo29nzrKNKc751KdbQ1Z+9xN/Hz7pZ6XBhw0asASEwkWKkC1bNmJjYhKtj42NJSws3OM+YeHhHsrHpFj+atdYxNEX462+2CTlYzLuHLo0ejpmeLhv59C1X7IyPnyPI8f/4b//LhJ6XeJ+hdDCBYg+6rmr4Mjxf+gx4m0KNxxBpY5jqdF9EnFn/uX3v44mK1uqWCFaRlbi3SXrvdLlTrCfQ0tirMEJQnLmzEnNWrVZt3Z1wrr4+HjWrV1Nvfr1Pe4TGdmAdWvXJFq3ZvUqIiM9l7/aNaakb+3a1USmom/tmuT6Uvo+6aVx7ZrkGuvVb+BZY/0Gib4TwOpVXxLplC9dpgzh4eGsdStz6tQpNm+KSihzpVz47yLbdh+kRWSlhHUiQot6Fdm08/dU9/33/H8cOnyS7NlD6NYqguXrdiYr06tLA2KPnWbFN7u80uVOsJ9Db7AutQxGRG4TkR9E5KyIHBWRVSKSV0TqisiXInJERE6KyFciUivJvhVE5GsROSciP4lImyTbSzsutltEZK2InBGRHSLSIEm5xiLyjaPhoIhME5G8btsHicg+5zgxIrLgcvpT+K65RKSAawFSDUd5aNjDzP6/d/jg/ffYs3s3w4YM5ExcHL169wVgQL8+jH3q8YTyg4YM5csvPufVKa/w8549PDNpPFu3fM8Dgy5FHx07dowdO7aze/dPAOzb+zM7dmxP0dd9OYJd49BhDzN71jt8MMfoG+rS18fR17cPY5+8pG/wQ4n1PT3R6HtwYBJ925Po2+77ORw6fASzZ719SeNgo7G3o7H/vb0Z465xyDC+WPk5U5NqdM6hiDB46HBeePZpln+yjB9/+IH+fXtTrHhxunT13ts87YM19O3ekLs7R1KpTBjTnuhJnmtyMWepCaR4Z1IvJj7UJaF83ao30LVlDUpfX5hGNcuxbMZgQkKEye+uSlSviNC7a30+XB7FxYvxXutyJ9jP4ZWSFYIGUNWALEAx4ALwMFAaqAYMAvIBLYF7gBuBysA7QDSQ39k3BPgBWAXUAJoCWwEFujllSjufdwOdgIrAfGA/kN0pUw74BxgOVAAaOvXMdrbXAf4D7gRuAGoCQy+nP4XvO97Rk2j5+/AJjfs33uPyypRpWrJUKc2ZM6fWqVtP132zIWFbk6bN9O5efRKVf/+juVqhQkXNmTOnVq5yky5csjzR9jff/r9kxwf0iafGpqjhckugNZ45n/ryytQk+r7dkLCtSdNmek+vPonKf+Cmr0qVm3TR0uWJts98J2V9KWk4e0FTXSZPnZ5I41ffbkzY5tLoXv6D/83TChUdjTfdpIuXfZpo+5nz8fr4k2M0LCxMc+XKpS1attKdu35OVUPuiMEpLsOfm6sHDh3Vc/+e1007f9cm97yYsO2rzXt1ztINCZ9b95+iP/16SM+eO6+Hj53WDz7ZqGXaPJGszk4PTldV1apdJqR6bNcSzOcw5uhJ131QwMdnYQFAP1q/V5fs/Nvr5aP1e9N0fH8uARuH47RYtgClVfXAZcqGACeAu1R1uYi0BT4FblDVQ06Z9sAKoLuqLhGR0sDvwABVneWUqQLsAiqr6h4ReQe4qKoPuB2rMfAVkBfoCMwGSqhqolAdb/Q75XMBudxW5Qf+TGkcjuXKyAzehMzg8khpHE6wkNo4nECTXuNwPl6/z+dxOHc0rODz8f1JIF1qO4DVwA8iMl9E7hORQgAiEiYibzuurJPAKUzLp5Szb2XgoMvYOKQUIO/uPP7b+T/U+b8GcK+I/ONagJWY81IG+BI4APwmIu+LyN0ikudy+j2hqv+q6inXApxOqazFYrFcjQTM4KjqRaAN0AH4CXgI+FlEygDvARHAMIybKwI4CuT04VDuo9BczTnX984HzHTqdy01MO61X51WTS2MS+1vYCKwQ0SuvYx+i8Vi8QrX9AS+LJmFgAYNqOE7VR2H6R85D3QHGgHTVPUzVd0F/AsUcdt1N1BSRIq5rfMllGgrUEVVf/GwnHc0/qeqq1R1FFAd01/T8jL6LRaLxSskDf8yCwGbnkBEIoFWwBdALBAJFMUYk31ALxH5HtOh9hJw1m33VcBe4D0RedQp84wPMl4ANorIDExgQhxQBWijqkNE5GagLPA1cBzTpxOCacmkpt9isVi8IitMwBbI+XBOYaLLhmMMxgFgpKquEJFo4C1MC+Qg8ATwsmtHVY0Xke7ALGATJvJsKPC5NwJUdaeINMMYq28wEYa/AnOdIieAWzARZrkxhvBOVd0lIpVT0u/VWbBYLJYsQsAMjqruBtqnsG0bUDfJ6gVJyuwFmiQpI27b97t/dtad8LBuM9A2BR3fAs291W+xWCzeIgghPrjHrEvNYrFYLF5hXWoWi8Vi8QtZweDYXGoWi8Vi8Qu2hWOxWCxBgK8hzrYPx2KxWCxeESJm8WW/zII1OBaLxRIEZIUWju3DsVgsFotfsC0ci8ViCQKyQpSaNTgWi8USBJjJ1HxxqWUerMGxWCyWIMAGDVgsFovFL9igAYvFYrFY0gnbwgkwISFCSBC3iQ+f+jfQElIld47gf2fKf02OQEu4LMc2TQ+0hFQpevd7gZaQInrh7OULXQE2aMBisVgsfkHwLQAgE9kba3AsFoslGAhBCPGhueLLlAaBIvj9ERaLxWK5KrAtHIvFYgkCrEvNYrFYLP4hC1gca3AsFoslCLDjcCwWi8ViSSdsC8disViCAR/H4WSiBo41OBaLxRIMZIEuHGtwLBaLJSjIAhbHGhyLxWIJAmzQgCWgvPn6a1QqX5pr8+WmScNINm/alGr5hQvmU6PqjVybLzd1Iqrx+YrPEm1XVSaOH0uZksUolP8aOrZrzS/79qVJ43vvvEnDiIpUKF6QLm2asH3L5hTL/rznJx7ocwcNIypSqnBu3nnTc/4ub+q8HLPeeoPaVStQsmh+2rdoxNbvU69r2eIFNKxdlZJF89Osfk1WrVyRrMzen3fTq2d3ypUoQunwa2nbrAF/HvzDZ43Bfp3ffOM1bqxQhkL5r6Fpo/ps3py6vkUL5hNRtTKF8l9D3ZrVk+lbsngRnTu2o0R4EfLkDGHH9u0+a3NxX9tK/Dj9Vg6/fw9rnu5I7XJFUi0/qGNltk7pRuz7d7P7tdt4rnddcrnl5evfphIbXuzMX7Pv5K/Zd7J6UgfaRFyfZp1ZHWtwgpT58+by2KMjePKpcWzYtJXq1WvQpVM7YmNjPZbfsH49fe65kz59+7Nx8zY6d+1Gj1u7sevHHxPKvPLyi7w+YxrTXnuTr7+LIm/evHTu1I5z5875pHHZ4vlMGjOK4Y8+yadrNlK5ajXuub0zRw571njuzBlKlS7D6LFPUzQsPF3qTI0lC+cx7olHeWT0U6z6JoqbqlWn5y2dOJxCXZuiNvBAv17c1bsvq7/dRIdOXehz123s/unSOfz9t1/p3LYF5StWYsmnX7J2/RZGPPYEuXLn9lofBP91XjBvLqMfHckTT41lfdQWqlWvTtdO7VPUt3HDevr0uos+ffuxYdNWbu7SlZ63dU+k70xcHA0aNmLSs897rccTtzQozXO96/L8wh00Hv0JPx44zuInWlOkgOdrcnujMky4szbPLdhBnRFLGDxzPbc2KM34O2ollDl0NI5xH22l6ePLafbEp3z1YzQfP9qCG0tcmy6aPeFK3unLklkQVQ20hiyJiBQATsYcPUmBAgWSbW/SMJLadeoyddoMAOLj4ylfpiQDBz/Eo6NGJyt/z109ORMXx6KlyxPWNW1Unxo1Ipj++puoKmVLFWfowyN5eMQjAJw8eZIbrg/jrVnv0qPnHR51ppYtukubJtSoWZtJL05N0BhZrTz33jeQwcMfTfX7N4yoSL8HH2LAgw+lqc7UskW3b9GIiFp1eP6VVxPqiqhclgEPDGLoiFHJyt93712ciTvDh/OXJKzr0LIxN1WvwctTXwPg/nvvJnuOHLz+9rupfj93UssWHSzXOaXnQNNG9aldpw5TXr2kr0LZUgwcNIRHPOjrddcdxJ2JY9GSTxLWNWvcgOo1ajD9tTcTlT2wfz+VK5Zlw6at1IiISPEcAYTeMyfFbWue7sjWX4/yyOwowDyA97x+OzM/383kpT8mK/9y30gq28EP6wAAIABJREFUXV+Qzk9/kbDu2V51qFO+CG3HfZ7icQ7MuoMxH3zPnLW/JFqvF84St3ggQEFVPZXqF/GA61nw1c6D5Muf/FlwOf45fYpm1Uv6fHx/Yls4Qcj58+fZtnULLVu1TlgXEhJCy5at2bRxg8d9ojZuoEXL1onWtWnbjiin/P7ffyc6OpqWbmUKFixI3XqRCWW81fjDjq00btYykcbGzVqwdXOU1/Wld53nz59nx/atNG2RuK6mzVvy/aaNHvf5flMUTZu3TLSueas2CeXj4+P58osVlCtfgR7dOlGl7PW0b9GIz5Yv9Uqbu8Zgvs4ufe7Hc+mL2uj5HEZFbaBly1aJ1rVu05ZNKZRPKzmyhVCzbGHW/XAoYZ0qrPvhEPUqFPWscW8sEWULJ7jdSofmo23N6/li218ey4eIcGvD0uTNlZ2ovYfT/0u4kDQsvhxOZLCI7BeRcyISJSL1Uil7n4h8IyLHnWVVauVTwhocN0RkvIik3aGcRo4cOcLFixcJDQ1LtD40LIzo6GiP+8RERxMalqR8aBgxMaa8a79kZcIulfGGY0eNxiKhoYnWFwkN43BsjNf1pXedrrqKFk38fYuGhhIb47mu2JhoiiY5dtHQsITyhw/HEvfPP0yf8hItW7dl7pJP6di5K33v7sH6b7/2Sh8E/3V26QtLdrzQFOuKiY72+H18uceuhMIFcpE9WwixJxO7C2NPniP02ms87jP/u995Zt42vpjYnmMf9uKH6bfyza4YXl7yQ6JyVUpey9/v3cXRD+9h6oAG3PXyWn7+62SGfA9/IyI9gcnABKAWsANYKSKhKezSHPgf0AJoABwEvhARrzq2bJRaYl4GgnsmKkvA0Ph4ANp37MyDQ4YBUK16BJujNvDerLdo2LhpIOVZrpDGVcJ4pHt1RsyKYvO+w5QLL8AL99Zl1PHqvLhoZ0K5fYdO0WjUJxTIk4Nu9Uszc3Bj2o//PMOMTjpEqeWXxB06/6pqSj7xEcDbqjobQEQeBDoB/YBknWuqeneiY4oMAG4FWgEp+zuTcFW1cEQkp4/7iYhkV9V/VPVoeuvyliJFipAtWzZik7zVx8bEEB7uubM9LDw82Zt7bGwMYU7nvGu/ZGViLpXxhusKG41HknQeH4mNoWiSN9xA1Omq6/DhxN/3cGxssrd/F6Fh4RxOcuzDsTEJ5a8rXITs2bNT8cbKicpUqHQjf/550Ct9EPzX2aUvJtnxYlOsKyw83OP38eUeuxKOnvqX/y7GE1owcYBAaMHcxJ7wPBPnmB41+fjrX3lvzT5+OniCTzb/wYT/bWNkt2qJOuAvXIznt5jTbP/9GOP/t5UfDhxjUMfKHutMD9IhaOBP4KTb8rjn40hOoDawyrVOVeOdzw2uUG4eIAdwzJvvGHCDIyK3icgPInJWRI46vsG8IrJORKYmKbtERN51+7xfRMaIyBwROQW8JSKlRURF5A4RWe/4J38UkWZu+zV3ynQQkS3Av0DjpC41p9wmEYkTkRMi8p2I3OC2vauIbHWO8ZuIjBORNLcac+bMSc1atVm7ZnXCuvj4eNauXU29+p7vh8j6DVi3dnWidatXfUmkU750mTKEh4ez1q3MqVOn2LwpKqGMtxqr1ajFd1+vTaTxu6/XUatupNf1pXedOXPmpEZELb5Zl7iub75aS5169T3uU6deJN98tSbRuq/Wrk4onzNnTiJq1eGXfXsTlfn1l32ULFnKK32u+oL5Orv0uR/PpS+yvudzGBnZgLVrEp/DNatXUS+F8mnlwsV4tv12lGbViiWsE4FmVYuxaZ/n/pZrcmUnPkmQxMV48zm1FkaICLmyZ0sH1Z5Jhy6cEkBBt+W5FA5VBMgGJPUtxwBX+mbwAnAIN6N1JQTUpSYixTB+wVHAYiA/0ATvusEeASZifJHuvAQMB37CNB8/EZEySVowzzv7/wYcx/gpXdqyA0uAt4E7gZxAPUCd7U0wTcmhwDdAOeAtZ/ekWhCRXEAut1X5U/tSQ4eP4L5+fahduw516tZjxrSpnImLo3efvgD0v7c3xa+/nknPmHtq8JBhtG3VjKlTXqFDh07Mn/cxW7d8z2tvvOU6PoOHDueFZ5+mfPkKlC5dhgnjx1CseHG6dO2WmpQUGTBoKCMHD6BaRC0iatVl1szpnDkTR4+7egMwfGA/wosVZ/TYpwHTCb3v593O3xeI+fsQu37YQd68+ShdttwV1ekNDw4ZxkMP9qdGzVrUqlOXma+buu64p485Z/f3pVjx4jw1/hkA7hv4EN06tOL16VNo064DixfMY8e2Lbwy7fWEOgcPG8H9995Ng0ZNaNSkGWtXfcEXKz5l8Wde/e4SCPbrPHTYw9zX/15q1XL0TTf6ejn6BvTtQ/HixZno0vfQUNq2as6rU16hvZu+Ga/PTKjz2LFjHPzjD/7+23T079v7M2BaRym17FJjxqc/MXNQY7b9epQtvx5hUMfK5MmVnffXmWiymYMb8/exM4z/31YAVmw5yJBOVdix/xjf7ztC2fD8PNUzghVbDiYYovF31uLL7X9x8Mg/5Mudgx6Ny9KkSjjdnv3Sa31+5LQ/otREZDRwB9BcVb2KtQ90H04xR8MiVT3grPsBzA/nClmjqq+4PohIaefPGaq60Fk3EGgP9AdedNt3rKp+6bave70FMG8Jy1X1V2fdbrft44DnVfU95/NvIjLGqT+ZwcE0b8dd6Ze6vUdPjhw+zMQJY4mJjqZ6jQiWLv88oQP34ME/CAm51EBt0LAh777/ERPGPcW4p56gfIUKzFu4hJuqVk0oM/KRUZyJi2PIwPs5ceIEDRs1Ztnyz8nt4xiSLt1v59iRI0x+fiKHY2OoUrUG789bluD+OvTXwUQaY6IP0aH5pZbKzBlTmDljCvUbNWHesi+vqE5v6HZrD44eOcKLz04kNiaaqtVq8PHC5Qmd2n/9mVhfvcgGvDlrDs9NGsezE8ZQtlx53vtoAZWrXDqHnTp346Wpr/HqKy/y5KiHKVehIv/3wVzqN2jktT4I/ut8W4+eHD5ymEkTxyXoW7J8RYr66jdoyLtzPmTCuDGMG/Mk5ctXYO6CxYn0fbp8GQ8M6Jfwufc9dwLwxFNjeWrseK81LtqwnyIFcvNkjwjCrr2GnfuPcctzqzjsBBKULJwXjb/Uonlx0U4UGNOzJsWvy8ORU+dYseVPJn68NaFM0QK5mTmoMeGFruHUmfP8+Mdxuj37JWt/+NtrfVeM/1LbHAEuAkl/VGFAqtEdIvIIMBporao7Uyvrcf9AjsMRkWzASkzLYSXwBbBAVY+LyDpgu6oOdyu/BDihqvc6n/djOr6ecStTGvgdaKaqX7utX+zs21dEmgNrgRKq+pdbmfFAN1WNcD7PxrRuvsQ0Heep6t/OtsNAPsyFc5ENyA3kVdUzSb6rpxbOnymNwwkWUhuHEwykNg4nWEhtHE6wEOzj8VIbhxNo0msczne7/vJ5HE6jm6736vgiEgVsUtWHnM8hwB+YF3WPI3JFZBTwJNBOVX2Kcw/or1VVLwJtgA4Y19dDwM8iUgaIJ7nt9vTLjUuDhFT3VdW+mE609UBPYK+IuJzR+TAtlgi3pRpQAUjWzFTVf1X1lGsBTqdBt8Viucrwc6aBycB9ItJHRCoDbwB5AVfU2hwRSegDEpHHgEmYKLb9IhLuLPm8OWjAXw/V8J2qjgNqAueB7sBhjMsNSGgNVfVci0cSeimd/pjaJHaJXam+bar6nKo2BH4E7nI2bQUqqeovHpZ4b49jsVgs/kJV53Kp/3s75oW5vaq6AglK4fb8BQZi+rEXAH+7LY94c9xABw1EYuK4vwBigUigKMYwxAGTRaQT8Cum49+bREaDRWSfU9fDQCHg/7zQVga4H1iGicaohGm9uNr2E4HlIvIH5iLEAzWAqqr6lBc6LRaLxe+zE6jqDGBGCtuaJ/lc2sfDJCLQQQOngKaYaLICwAFgpKquEJEcmAf4HOA/YAqm3+VKGe0sEcAvQBdVPeLF/meAG4E+QGGMNX8NmAmgqitF5GZgLPAYcAHYA7zjxTEsFovFYOfDyVhUdTcmeszTtgvAIGdJaf/SqVS/W1U9Dt5Q1XV4uEyqOh4Y7/wdg3HtpYiqrsQEO1gsFkuayArz4QS6hWOxWCwWfA8AyEzTEwQ8aMBisVgsWYOrroWjqvvJVF5Ni8ViyRJdOFefwbFYLJZMSRawONbgWCwWSxCQFYIGbB+OxWKxWPyCbeFYLBZLEJAVotSswbFYLJYgIAt04ViDY7FYLEFBFrA4tg/HYrFYLH7BtnAsFoslCMgKUWrW4FgsFksw4OvcNpnH3liDE2hOn8rwKcjTxOkgn/HzQiaY8VMv2Bk/04peOBtoCSmSXtqyQBeONTgBJD9A+TIlA63DYrGkD/kxU65YUsAanMBxCChB+k41nR/4MwPqTS+CXR8Ev8Zg1wfBrzEj9OXH/KZ9Jws0cazBCRBqfBh/pWedcskBfFpVg+5NK9j1QfBrDHZ9EPwaM0hfmuuxQQMWi8Vi8Qs204DFYrFY/EIW8KjZgZ9XGf8CE5z/g5Fg1wfBrzHY9UHwawx2fVctEuzhkBaLxXI1IyIFgJM7f48hf/4CXu9/+vQpqpcJAygYjH1m7liXmsVisQQBNmjAYrFYLH5B8DFoIN2VZBy2D8disVgsfsG2cCwWiyUIyApRatbgWCwWSxBgx+FYLH5GREJUNT6ra7BkRa7+No7tw8miiEgzEckfBDrE+b8mQDA86F0aROQhESnl/J15ftVXAfZ8X51Yg5MFEZFngMlAWKC1qKqKSEdgi4i0DLQeFyKSAxgCjIGE3HeWDMLtxeMmEbk22M63iHh8Vqa03rdj+L5kFqzByWKISFmgBjBSVX8JAj2lgJbAYFVdE2g9LlT1AvAWUF5EikJwvnWnpikY9XpCRMR58egGrAAGiUjuQOty4e5iFZEmItJVRDqJSHZVjU8voyNpWDIL1uBkIURkBPApUBAIBmNTA3gHaAfsdNb5/feTygNjLhAB3AXB18pxe1C3FJHJIrJYRAaLSAkIPr0p4XyHm4GPgKeBD1X1XIBlJeBmbF4A3gaeB0YDP4hIofRyA9sWjuVqYxlwLdAIqBhgLWC0CFAeqAQJDx+//oTcHijdRaSz2/o/gZeB20Qk6GbKc85Vd2Ax5lxuBV4BJruMTmZARPICDwIvqOpbQIyIXC8iQ0WkuYgE3PUrIoOBfkAvVa0MLMDcsw3cyqTpvpU0/MssWIOTRXDehn/B/ECOAmNEJKBGR1W/Ap4C1gAPiUgXZ71fjY4YwjFvri+IyLci0lZEQoH5mIm6Kjplg+Y34xjBScDjqtoP0zo4B+x3jGVmIRdQGjgvIgWBZzCtnbHAh8BtEDgXoXPcKsCzqrrZcf1NAh5Q1c9EJK+IZMssLcpAEjQ/HkvGICJdRGQYxi9eU1X3Y4xOdeBVEangJx2uTuFiIlLO9daqqlHAC8B+4GHHtZLhRsfdcKghGmgK3AIcB8YD6zCtrz+BJ0UkZzBE0bkRApwB3haRchid81R1FICI1A6kuJRwuxcqi0gBVT0GvI855/uBssAcVS2CuQbtwH8uwqT3nXPckkAOEengaH1MVd927qN+wH1pP3AalkyCNThXMSLyIjAV6ILpmN8iIm2dlk5doA4wVUQqZ7AO907hZcB3wPsi8jSAqn4NvAqcAIaJyC3O+gx5wCTpBI4UkfYiUh04o6p7VLUzMAzThzMNKI5xQ9Z27Z8Ruq4UEbnG+TMfRls7YCWmf26gU6Y6MF5EIgIiMgXc7oWumHthhBMR+BLmHu2NadG86+xyGvjLKeMPfSGu+05EbnC71lFAd+BjjLF5w1lfGGiPmWI6bcdOw5JZsAbnKkVE7gR6AXeoaitgibMpFEBVfwPqAx2AARmpxXnAdAA+wLhKWgLbgYEi8oZTZh0wBXNP3isi+dJbh+M6kySdwIuAN4DNwFsi0t7Rs1lVJwA3Aw9j5qsf4mwLWCtHRGoBu0QkVFV3YQzNIuAHVb1fVS86RXsCRYGYAEn1iHMvdME8uF8G3lPVC6oar6rfqeonznco5byQ3AHMcKIGM5QkLyLjgTmYlzIwrZr8mPO5RUTyiImwfA9jdKak/fhXf9CAzTRw9VIeWKiqm5wWw+sYn/MHYubfuE5Vf3X6AaIzUoiIFMdE9Typqq+KSCFM5NduoKWIvKGqA1X1axEZCxxQ1X/SWUMJ934NEbkf6It5m96JacE8BAwRkThV/QZAVfcAe0TkDPCaiFRR1Z/SU5uXXMC0BFtiHtrzMP0fxcSMZwpxtvUHmqjq3wHS6RERuRZjuMep6kwRye30n3UDtgG7MP0lj2Lcvs0dw5rRutxfRJ4D7gWGAgcBVPUPp4X+GTALY8x/xZzvJqr6n9OPc9FT/RaDNThXES53hfMxO5DNiWJ6D3hUVd92tnUFbhSRF1T1L2ff7Kr6X0boUtVDIrIYWO303azDuFMeBWYCfUUkv6reo6rfpffxReQ1IA4Y5fZQiARWOO48gE9F5B9MlFdn4BtJnOLmNyAHEOjxIbsx/RwPAB+r6koRyQXcjomc2ocJCmmiqjsDpjJlFCgFXBSRnMBEoCEmKCM/5kH/OTAb2KWqBzJSjIjUUNUdbm60+piXoR6q+o2I5HIMYk3gG6AWUM/Ruxf4SlUvpsfvJyvMh2NdalcXDdz+/hXzpvs+JorpTcA1u+CdQHZ1mx0wo4yNW/1TVfVHzANlLzBGVeMwb7V7gaJOSygj+AJ40vn7Wrf1+eFSn4wTNfc/oL+IFEziOmuCaUkcySCNyXDrXE/ov3Cu0wjgJqeVhqouU9VeQGWgMdA1SI0NqnoSE/k3BnMuKwIfqGoosBzoqaonVfUzPxibp3HuC7dAgYKYqad/FJF6GIPoui8WA6VUdY2qvun8f9F5iUn77ycLdOJYg3OV4HQOfytmvACqOgfTLwFwTEQqikhVjAsmjOQ/tPTQIG4PySpOZ3xbESnvVqwiUFRVjzqfizuaeqjqofTS4tIDoKpLVfWCiPQGPnQM2wqgm4g0TmJYDmIMYLxbPTmAf4CqqvpHempMDae/oxWwXET6Oy0ZMP1Jy4CGzht4iNO6PaCqp1X1tL80pobbvRAhIneKSD8RKaWqYzCBLP0xLbO3nF3+AQ74MShjIc6gXkwUGpixTCUwLymrgEKY0P12mFZO2aSVpJcbLQvYG+tSuxoQkUHAjZgxGNNE5BpVfVlV7xGRpZj+k8rA95i3t/rp6XN23GGn3dwStwAzgN+B64CjIjJLVWcD64EIEfkI4+bqCdR23nzTFQ9RbnmBApiIqEcxEWjLReQuTN/BSUwAxRHMw89VzwURWeSvsNwkHMC4oQYAj4nIkxiX5OvAJuD/3NyCQYVjMG/FREr+CZzFBGbcrqqLXeVEpJTTWusKJH0ByEh925zjd8cMEeirqqudF7M7MYEtX6vqaRHJhvEa+CVa7mrFGpxMjuMWuA8TSbUTaA6ME5EcqvqcqnZ1fkDFgL+APWryP6VLn42IvIXpK7rfcS/Uw6T/GKOqr4uJTluGaVGAcZuEAa2A/zAPmH1p1XElqOobTud/f0xU0cuYh+ACTL/HKeA8UM95WCb0ifnL2Lgf07lGv4jJflAKGIVxRT2BSb76BaZfakdGGOy0IiaibibGpfu209LdiwkGWOyUaY4xpvWBlv4IyEhyjqtjrvkm4EURGamq60TkWeceyCUihTERliGYezmDdPkWcZaZotQkMC9tlvTA6YD/FJiuqu8560pgDNCjwBOqOtXDfuky34uI3IFpJbRze1vsD9yqqh1FpDSwFvhcVV3jQwq73GkikkdVz6RVxxVqdX/I9AX6YKLzBmLcKa6sC4vTqxPYV42OG81lZNYAn6kJY0dEGmL6aUZhWo97gcggNTjdgXtU9VYRKQN8DXyiqoOc7fkxL70tgK1qBiVntCb30OepGFdZE4wHYChQARiqJmIyJyY4w+V2a+q0dtM1Gs3pVz35+6FjFChQwOv9T506RZni1wEUdO+XDUZsH07m5iJwA1DEtUJN6O87wA5MTq1hrm1ufRrp5bIoCRxV1W1iMugOx9xTB53Inm8x40QGO8dvA/QTExaNv4yNcyx1+/6zMZF712PG4JxW1QWYMPL06wT2Ajdj0x3zElEcEwI9CXhJRFyj7der6ouYQagjgC7BaGwcigHFxWSzWIcJKR4CICajxAvAv6q6yB/GBhLlzSuE6Z8ZoqpH1ITBv4ox4NNEpImqnscYyXmYlvgF50UkQ0Kfs8I4HGtwMjcngU+ASHFLUaOqBzGdn6uBkWIGgWaEW2gd5lm5GuMiOYDp/+gN/AgsUtUH3QzcbUA1jAvD73gwOv+HebA/JyJl3NxnfhlLISIdHZeOS9v1wARMCHsPVe0JtMW4IAeIyA3OftnURHC9qqp7/aHVRzZirnUUsEZVH3Db1hIzCNnvbn0ReQDTH3Mjpp8RAFX9FtNi34vJwNFKTcj0lEC9iFxtWIOTyRATbVYFEuZs+RzjE79PRCo5ZfJj3i7nARuATo4vOl3fhVR1M8aotQA2qupipzP4Lczb4zIRKSgihUXkeUxqkOfUhEMHBA9GZxbmXD2QEecoJRx36AxguFxKLXQBE9jwp1MmxDnHD2PSpzRzdF90fRd/aL0crnMmIjXERCU2dTZtx/QrXgC2i0nMeb2YgZW9MYM/A+EC2gL8BNyEM65KnNBzx+i40iz1ct/JXy8iVzO2DycT4fZDzYZ5M+vldCoPwOT++hfzsCqBGWcTISIvYZJSNkzvH4yYnF7LMYMiGwI7VfVOEcmDaT10w4QZH8E81Lu7+noCTZI+nZcwndatHDeKvzTUAt7EuD9fBf4A9gATVfVNpw/hPyfIYyUmC/QDKdcYOBxX4AeY610RmI5x+YHJi1YDk/1iB8YF3MMf94Kn/kon4qwqJhP1eYy77Ix7v53T8vwxHd3PqWksAJw8EO17H84N4ZmjD8dGqWUSnB90T2AQ5kcyBvhSRG5V1XdE5GeMX78BJnppgrNrKOZtLhumzyfdUNWzItLZ+bH2w0RMzVHV3sAdYnJmXQccw3QKB03K/CRRaP9gXGvX4Ed3n6puddw77wDDMen4X8KE6O5y+hVcZAOCJk2NWytRnSiuxzB9deswLe65mFZuX8xL0o2YUfq/AAc1ncdcpaDRPUCglaPnAPCzqu5wgl4WAutEpJlzP+dQk9ttZ9I6MlxvFsg0YFs4mQDnh3EdkE1VpzvrcmDcWaWAW1R1a5J9SmCM00DMG1yG5qMSk2zzdsyDZ6uq3nWZXYIC58F5G7BXVXcESENNTIvwe8yI9q6YazcaY6yrAPdjwrV/DoRGF2IGzUa7PcjbYSK9rgOGq+oJZ31zTCj8PEzUV8ACG8QkaR0IxGJ+L8uAt9WkBarmaDyBaeH6LZDFTV8B4OTBmOM+t3BKhhWCTNDCsX04QY7THzMZ4+8v4awTp/+mFeaNba6INHTzpecDHseE1rbIaGMDoCbZ5jxM5FE1Ecmw8QrpiRrmB8rYOBq2YeZUqQn0wGR/Ho55SI7AtFqbBYGx6YdJRRTp1tdVDKM1IUW/0ypYh8lE3g14R0Su86NOcfu7HsaAd8S0vNoDeTBzLzVT1R8wnoOKmIABSwZiWziZADEZnedhRsnfrKq/u4XRZgd+wKSn7+G2T2Egp/o5W7CY6YJ7Y3KmdfeH6+RqwenTeQvzUB+Dad3kwLRsA/7m6jzId2CyqdwHbHait27DtMxexgz4/c/t/myDCUGvHYB7cRQQDuRR1Qfd1jfCJGndpKpDxaTSKQv8HojAAFcL5880tHBKZJIWjjU4QYqItMZMsBWvqsscF9kKzMj4W1X1oNuPOhtciqJx7xAPkPY8QI5AulEyK457bSYmEGOiBnYqhATEzHZ63vl7C5ATeBATnXhRTJ66/wOeBSY461z35zWqetYPGt37bAphXJKPYlyVbdzvRxF5EGN0yqpqjNt6v08xkGBwYtNgcEIzh8GxLrUgxIlGexfTiTxXRN51NnXEuAMWiJnfJWHciGucgPM5oG8RqnrGGhvfcNxrgzFv5scDLMedCwBiskc8gQkpfgGo5zyk52Dcgk8AY5yIL9f9meHGxjmOy9g8CzyHGTQ7HhNMc4vr9+FwAGPUJUkdAQt9ljT8yyxYgxNkOG6APphAgFqYN7TemLBZxfigc2Pmawl139eOE7g6UDP2pr2/XVCp4bRUumHm42mMiUIrjhnH5G507sW8KI3yl7YkfTbtMOO93lHVf1R1IiZ56JvAYDGZq0th+p1OEGQzol7t2LDoIMKJAKoCPKyXZuqcCDyNyfP0KjAS0wk6CZNw0nIVoqrnAq3BHREpgmk1PK2qzzjrrsPkepuFmUNos5oZZS9gBnz6BbfxVD0x46mWq+r3rnE1qjpSROIxhucMpr8pGyZZqPoz9Dk1skLyTtvCCS6OAUuBlSJSB+NjHq+qYzGGpzvGT35GVXu5u9EslgzmP4z7aR+YsHxVPQa4+hqfARo7LZ25qro7owW5RWWGOMEzj2AGQFcFM1mdXJpc71FgHMYlvVpVW+ul3GgBNzZAlpgPxxqcIMJ5q13ujGVojZmj5T1n83nMyOh/cZt10rrRLP7AuSfjMaH4rjmCsmNekn7ATIvxHH6cL8atrzJUTYaApsASoKqI3O0EOsS7GZ1JGC/Bu2Lm6cnwmW69ws8WR0QGi8h+ETknIlFOCHlq5W8XkT1O+R9EpKO3x7QGJ/hw/QAqYqa7VRHJjRlct1xVO7j/iCyW9Ma9TyQJT2Py8j0O5mHttA72YPp17vS3K1BEegGzRKSuE5xwN6af6WHgZqcl5m50Hsak3ZkvIl39qTWYcNyPkzEZSWphwt1XJu0XdivfEOOKnIUZL7YEWCJmrq0rP64Niw5ORKQ+JjX6z0AuzGyetYLqjcxy1eEWytzW2TmSAAAJ4klEQVQUkx+vFCb1zo+YYBXXgNQvMLO31sQ85G9U1b8CoLcvJgvDr8BUp+8mDyabQAFMq2u5M1Dafb9ngPdVdY+/NSfFFRYdfeSkz2HR4UUKghdh0SIShRlH5ZouIgSTB2+6qj7vofxcIK+q3uy2biOw3X2M0+WwQQNBiqpudIzOLZiZKCc7Pmm/TwxmyTo4xsbVV/gdxsh8DjyPMTyvYFy9T2FCo/8DmvjD2Hjq3FfV2SJyFhNKPlJEXnGMThfMlBlTMW6/r5Ls92RG6/WW06dP+RQAcPp0go3Jn6Rx+q+q/pu0vJiksLUxxhgwIeUisgqT1cITDTAtIndWYjJJXDHW4AQxavKjJeRIs8bGktE4LzkzgBHOwzw7ZrDxSExy0zdUdSGwUEy28BD103QTbuNs2gC/qeqvzvqPnTf0gcAjIvKcmuSct2CCGb71h740cB6IrlCmZHga6vgHZ1oLNyZgxiElpQgmSi9pSHgMJsmqJ8JTKO+VZmtwMhHW2Fj8QDmMq2m2mGmh1wCvA3GYB9h/IvKxqh7w14DOJBkEIjD9CEud1sx+AFX9yHlznwbEi8gMVV2PcQEGJIPAlaKq55xznTOdq07Wugk01uBYLFkYtz6bGsBhzPQC25xAlZmYEOJhTtnemHQx50Vkmj8e4EmMTRdMv+bLmMnRHhaRKW5G510x05w3wfR9rnd9v2A1Ni6cYAt/BVwcwUxVEpZkfRgQncI+0V6W94iNdLJYsihuxqYbJk/fg8BxNfnbimHcJQudstcDa4G3gU/8ZGwkSbqat4A7VHUaJmKqKWbG1NJOmXBgM6Z/aRIEPs1TMKImJ94WnBB3SAgaaIWZIdgTG9zLO7RJpbxHbAvHYsmiOMamE/ARJpPFZ3ppPph8QGGgqIjcgElZUwq434+50VwZBMZgslN3xBl4qqqTnWCBXsBrIrIGaOvsOieYMggEKZOB90Tke2ATxvWYF5gNICJzgL9U9XGn/KvAVyIyEvgUuAOog4kQvGKswbFYsiiO26wPMEXNrLF5RKQsZiK9zZiZYidjkogWxOR384uxcdN4HU5LRlU3i8j1YjJq3wGswkxxXgVjEH/BTF2t7q0jS3JUda6IFMVkMAkHtmOuryswoBRmoK+r/HoRuQszFutZjOHvpqo/enNcOw7HYsmiOFFmX2PcIuMxQQHVgEqY/oRXMFmVBdjp6ivxs8ZCmDFAszFjfwYBZTDdASUwrrOZGIN43DE2NpozSLEGx2LJwjiBAG9iph9YDSxR1TkiMh1jeNoHuqUgIv2BlzChvG8CX6rqKhH5ALioqn3cylo3WhBjXWoWSxbGMS7fA9er6pduKZMEE4GUgwCH16rqLBH5Esilqq7koSEYV9DGJGWtsQlibAvHYrEkICI3YjriBwONvfXRZzQikg+IAB4DbsCme8pU2BaOxWIBQERqYzIKRADNgtDYCCYyaiSm5VXbSfcUtIM6LYmxLRyLxQIkBBHUAfar6sFA6/GEiOTCRKXtcPJ/2QCBTIQ1OBaLJVNiAwQyH9bgWCwWi8Uv2NQ2FovFYvEL1uBYLBaLxS9Yg2OxWCwWv2ANjsVisVj8gjU4FovFYvEL1uBYLBaLxS9Yg2PJ8ohIaRFRZ/piRKS58/naAGhZJyJTU9k+XkS2e1mna5K1tOh6V0SWpKUOi8UaHEtQ4jzg1FnOi8gvIjJWRPyRjmk9ZsbLk1dS+HJGwmKxGGwuNUsw8znQF8iFme3xNUwa/eeSFhSRbJhJItM88tyZgterudotFsvlsS0cSzDzr6pGq+oBVX0DM8NjFwARuVdETohIFxH5CZNCv5SzbYCI7BaRcyKyR0QGuVcqIvVEZJuz/XugZpLtyVxqItLIacmcEZHjIrJSRAqJyLtAM2CYW4ustLNPVRFZISL/iEiMiLwvIkXc6swrInOc7X870/d6hYjUFZEvReSIiJwUka9EpJaHosUcLWdF5DcRuS1JPSVFZJ5zTo+JyFLX97BY0gtrcCyZibNATrfPeTBp6gcANwGxInI3ZtrcJ4HKwBPAJBHpAwnp7Zdjpk+ujZnp8uXUDur07ax29mkANAY+wUwINgwzY+bbGDdcMeCgY6zWANswCTHbA2HAPLeqX8IYq65AW6A54MlYpEZ+4D1HU33M1L+fiUj+JOUmAQuBGsCHwMciUtn5fjmAlcBpoAnQCPgH+FxEcmKxpBPWpWYJepy09K2AdsB0t005gEGqusOt7ARgpKouclb9LiJVgAcwD+a7MC9a/VX1HLBLREoAb6QiYRTwvaq6t5R2uR3zPHBGVaPd1g0BtqnqE27r+mGMUUXgENAfuEdVVzvb+wB/Xsk5caGqa9w/i8j9wAmMIVvutmm+qr7j/D1GRNoAD2GmbO6JOScD1EmuKCJ9nXqaY6Z2tljSjDU4lmDmZhH5B2NYQoCPMC0SF+eBna4PIpIXKAfMEpG33cpl51IAQGVgp2NsXGy4jI4IYL6X2mvw/+3dT4jMYRzH8feHy9Jelo0TxWpDlhO7JYqLUnKQkMjfw14cRHLaFDlw0CYXbf613JSLk5KojULZWvmXP4eVJQeWLZfH4fuMZtdsY+z0a9s+r8s08/zmN89vDvOd7/f5zjywLs9/rBZgBpGtPSw9mFL6KulFLS8iaS5wkggMc4isaya5vFhm7DX2EddVmusi4HvE9j8a8lzN6sIBxyazu0AnEVgGK+x7MpJG/915Y749SNkHeTaRDbpG/uM5jUTZ7ViFsY/EB3w9XAFmE6W998RaVh+jS4/VNAKPgZ0Vxj5PdIJmJQ44Npn9SCm9/teDU0qfJA0CC1NKveMc9hzYJamhLMvpqHLqZ0RJr2uc8V9EZlHuCbCF2Mzsrw3CJL0hOu7agQ/5sSagFbhXZT7lVhNlxdv5HPOA5grHdQBXx9x/WjbXbcBQSulbDa9tVhM3DdhU0wUcl3RIUqukNkl7JR3O49eBBFyUtFTSRuBIlXOeBlZKuiBpuaTFkjrLOs7eAe35B6TNkqYRLdyzgBu5k6xF0gZJlxRbIg8DPcAZSeslLQMuA7W2db8iAugSSe1EQ0CljGyrpH35PTkBrALO57Fe4AtwS9IaSQtyp153Xt8yqwsHHJtS8sL4AeL3O/1EtrAHeJvHh4FNQBvxDf8Ulcte5ed8SXSRrQAeESWrzUApczlLlOwGiBLU/JTSIJF9TCcW3fuBc8RCfCmoHAXuE6W3O8ADorRVi/1AE5GlXAO6gaEKx3UB24lsbTewI6U0kK/vJ7CWyLRuEllgD7GG44zH6sY7fpqZWSGc4ZiZWSEccMzMrBAOOGZmVggHHDMzK4QDjpmZFcIBx8zMCuGAY2ZmhXDAMTOzQjjgmJlZIRxwzMysEA44ZmZWiN81hKb50gi8/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 400x400 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import helpers.evaluate as ev\n",
    "evaluator = ev.Evaluate()\n",
    "import pandas as pd\n",
    "\n",
    "final_predictions = []\n",
    "\n",
    "for p in all_predictions:\n",
    "    for sub_p in p:\n",
    "        final_predictions.append(sub_p.detach().numpy())\n",
    "        \n",
    "predictions = [np.argmax(p).item() for p in final_predictions]\n",
    "targets = [np.argmax(t).item() for t in y_raw]\n",
    "correct_predictions = float(np.sum(predictions == targets))\n",
    "\n",
    "# predictions\n",
    "predictions_human_readable = ((x_raw, predictions))\n",
    "# actual targets\n",
    "target_human_readable = ((x_raw,  targets))\n",
    "\n",
    "emotion_dict = {0: 'anger', 1: 'fear', 2: 'joy', 3: 'love', 4: 'sadness', 5: 'surprise'}\n",
    "\n",
    "# convert results into dataframe\n",
    "model_test_result = pd.DataFrame(predictions_human_readable[1],columns=[\"emotion\"])\n",
    "test = pd.DataFrame(target_human_readable[1], columns=[\"emotion\"])\n",
    "\n",
    "model_test_result.emotion = model_test_result.emotion.map(lambda x: emotion_dict[int(float(x))])\n",
    "test.emotion = test.emotion.map(lambda x: emotion_dict[int(x)])\n",
    "\n",
    "evaluator.evaluate_class(model_test_result.emotion, test.emotion );"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Words\n",
    "You have learned how to perform neural-based emotion recognition using RNNs. There are many things you can do after you have completed this tutorial. You can attempt the exercises outlined in the \"Outline\" section of this notebook. You can also try other types of neural architectures such as LSTMs, Bi-LSTMS, attentions models, and CNNs. In addition, you can also store the models and conduct transfer learning to other emotion-related tasks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "- [Deep Learning for NLP](https://docs.google.com/presentation/d/1cf2H1qMvP1rdKUF5000ifOIRv1_b0bvj0ZTVL7-RaVE/edit?usp=sharing)\n",
    "- [PyTorch Autograd Tutorial](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)\n",
    "- [A Simple Neural Network from Scratch with PyTorch and Google Colab](https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0)\n",
    "- [Building RNNs is Fun with PyTorch and Google Colab](https://medium.com/dair-ai/building-rnns-is-fun-with-pytorch-and-google-colab-3903ea9a3a79?source=collection_home---4------2---------------------)\n",
    "- [Deep Learning for NLP: An Overview of Recent Trends](https://medium.com/dair-ai/deep-learning-for-nlp-an-overview-of-recent-trends-d0d8f40a776d)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
